PaperlayerView exact revision on arXiv

Source-linked AI summary

Toward Native Multimodal Modeling: A Roadmap

Siyu An, Junru Lu, Junnan Dong, Qiufeng Wang, Yinghui Li, Weizhi Fei, Zichao Yu, Zheng Yuan, Biao Liu, Haopeng Wang, Renzhao Liang, Yixuan Yang, Yunhang Shen, Bo Ke, Keyu Chen, Linhao Luo, Difan Zou, Xiao Huang, Di Yin, Ruizhi Qiao, Xing Sun

arXiv:2605.25343v1cs.CV

TL;DR

Native multimodal modeling lacks a sufficiently defined architectural design space despite its importance for unified cross-modal understanding and generation. This paper formalizes native architectures and presents a roadmap spanning model design, data, training, deployment, and evaluation.

  • Problem

    Native multimodal modeling lacks a sufficiently defined architectural design space for integrating modalities toward unified understanding and generation.

  • Method

    The paper defines nativity through integration depth and input-output duality, then analyzes NMM across architecture, data, training, deployment, and evaluation.

  • Results

    The paper delivers a formal taxonomy distinguishing mid-fusion and early-fusion and organizes native models as Multi-to-Text, Multi-to-Target, and Multi-to-Multi.

  • Takeaways & Limitations

    NMM is framed as intrinsically multimodal architecture whose development requires coordinated advances across the full modeling lifecycle.

  • Takeaways & Limitations

    Existing benchmarks rarely jointly assess understanding, generation, and interaction, while accuracy-only metrics omit response timing, compute, streaming, and interruption handling.

Abstract

from arXiv · show

Multimodal modeling represents a vital step from modality-agnostic reasoning toward world modeling. While early approaches predominantly rely on late-fusion that assembles encoders and frozen language backbones with output heads, recent efforts have shifted the paradigm toward native multimodal modeling (NMM) with the intrinsic integration of modalities for superior multimodal performance. Despite its potential, the design space of native architectures remains insufficiently defined. In this paper, we present the community with a formalized roadmap for this transition. Specifically, we formally define the architectural nativity, distinguishing mid-fusion and early-fusion from non-native paradigms. We further organize the existing native models through the lens of input-output duality into three categories: (i) Multi-to-Text for cross-modal comprehension with text-only output; (ii) Multi-to-Target for scenario-oriented generation, e.g., image, audio and video generation, and (iii) Multi-to-Multi for unified modeling with symmetric input-output. We deliver a comprehensive and industrial-grade investigation into the transition toward the definitive NMM framework, where understanding and generation seamlessly coexist within a unified transformer paradigm. We systematically unpack the end-to-end pipeline from industrial perspectives from architectural coordination, massive data curation, to full-stack training recipes, inference & deployment, and the comprehensive evaluation for truly native modeling.

1 Introduction

The introduction frames native multimodal modeling as a transition beyond text-only, late-integrated systems toward intrinsically unified multimodal architectures. It formalizes this landscape through integration depth and input-output duality, then presents a lifecycle roadmap and future outlook.

  • Motivation: Text-only interfaces limit LLMs’ grounding in the rich sensory signals of real-world environments, motivating multimodal modeling for progress toward artificial general intelligence.LLMs show strong comprehension and reasoning, but their interaction with users and the world remains fundamentally text-based.
  • Problem: Native multimodal modeling intrinsically integrates multiple modalities through joint backbones or unified transformer spaces for more principled cross-modal intelligence.This paradigm shift responds to limitations of earlier approaches and seeks to internalize multimodal capabilities within the core architecture.
  • Problem Formalization: The paper formalizes NMM using integration depth, distinguishing mid-fusion from early-fusion, and input-output duality, distinguishing Multi-to-Text, Multi-to-Target, and Multi-to-Multi.The taxonomy clarifies a fragmented design space by combining structural integration regimes with modality-flow categories.
  • Technological Roadmap: The roadmap analyzes NMM across architecture, data curricula, training, inference deployment, and holistic evaluation, while identifying modal bottlenecks and cross-cutting technical solutions.It presents a full-lifecycle analysis spanning architectural designs (§3), data curricula (§4), training strategies (§5), inference deployment (§6), and evaluation (§7).
  • Future Outlook: The paper uses empirical insights from state-of-the-art implementations and paradigms to project future trajectories and strategic directions for advanced NMM.This future outlook complements the formal taxonomy and technological roadmap.

2 Task Formalization

The paper defines native multimodal modeling by the degree of intrinsic architectural integration, excluding post-hoc alignment schemes. It further organizes native models by input-output modality flow into Multi-to-Text, Multi-to-Target, and Multi-to-Multi paradigms.

  • 2.1 What is Native? Formalizing Cross-modal Fusion Nativity: Late fusion assembles modular encoders and a backbone that remains blind to raw sensory signals, relying on a grafted output head.The formulation is Flate = G.
  • 2.1 What is Native? Formalizing Cross-modal Fusion Nativity: Native multimodal modeling excludes post-hoc alignment and requires multimodal synergy to be an intrinsic architectural property.NMM is categorized into Mid-Fusion and Early-Fusion regimes.
  • 2.1 What is Native? Formalizing Cross-modal Fusion Nativity: Mid-Fusion injects modality-specific features into intermediate layers of a joint multimodal backbone through cross-modal alignment or injection operators.It is defined as Fmid = Backbone(C(E1(m1), . . . , En(mn))).
  • 2.1 What is Native? Formalizing Cross-modal Fusion Nativity: Early-Fusion maps all modalities into one shared embedding space from the outset with a unified tokenization operator and transformer.Bypassing independent frozen encoders yields a born-native architecture that treats modalities as equivalent tokens.
  • 2.2 How Native? Taxonomy by Architectural Symmetry: The taxonomy bounds native capability by input-output modality flow and architectural symmetry, producing three progressive paradigms.These paradigms are organized through modality duality and structural symmetry.
  • 2.2 How Native? Taxonomy by Architectural Symmetry: Multi-to-Text maps arbitrary interleaved multimodal streams into text after dense reasoning over multimodal hidden states.It is formalized as FM2T : M →T and uses either Mid-Fusion or Early-Fusion architectures.
  • 2.2 How Native? Taxonomy by Architectural Symmetry: Multi-to-Target directly decodes a single non-textual target modality from native hidden representations, preserving semantic coherence with the multimodal prompt.It is formalized as FM2G : M →yk, where yk ∈M.
  • 2.2 How Native? Taxonomy by Architectural Symmetry: Multi-to-Multi establishes symmetric multimodal input-output flow, unifying understanding and token-level next-step generation in a single world modeler.It is formalized as FM2M : Min →Mout, with arbitrary modality combinations permitted on both sides.

3 Model Architecture

This section examines three NMM functional paradigms defined by input-output modality configurations and outlines their technical challenges and approaches. It distinguishes this functional taxonomy from the orthogonal architectural taxonomy of mid-fusion versus early-fusion, annotating architectures accordingly.

  • NMM systems assign distinct functional roles to comprehension and generation across modalities.The section investigates three paradigms organized by input-output modality configurations.
  • The three functional paradigms are defined by input-output modality configurations, while mid-fusion and early-fusion describe cross-modal integration depth.These are orthogonal dimensions rather than competing taxonomies.
  • Each functional category includes representatives of both fusion paradigms, with architectures annotated as Mid-fusion or Early-fusion.

3.1 M2T Unimodal Generation

M2T models convert multimodal inputs into text for complex comprehension and reasoning. Their development centers on unifying modalities and addressing modality-specific challenges in vision, audio, and video understanding.

  • 3.1 M2T Unimodal Generation: M2T models process text, image, audio, and video inputs to produce text-only outputs for complex comprehension and reasoning.This design efficiently converts real-world signals into semantic representations.
  • Vision-Text Comprehension: Vision-text comprehension faces three core challenges: modality unification, multi-image reasoning, and multi-scale encoding.Native unification can create architectural tensions and modality competition, while long visual sequences risk attention saturation and quadratic computation.
  • Audio Understanding: Audio understanding requires end-to-end processing of waveforms or acoustic features while resolving semantic-acoustic conflict and high latency and computation.MiMo-V2.5 separates semantic and acoustic refinement in RVQ layers, while Gemma-4-E4B compresses audio representations to reduce forward-propagation cost.
  • Video Understanding: Video input expands representation from H × W to T × H × W, creating computational explosion, temporal and logical inconsistency, and long-range dependency challenges.These difficulties arise from redundant per-second tokens, temporal hallucinations, inconsistent object identities, and context-window exhaustion during long video processing.

3.2 M2G Scenario-based Generation

Scenario-based native generation requires joint multimodal modeling to address modality-specific challenges in image, audio, and video generation. The section highlights visual fidelity and controllability, semantic-prosody alignment and latency, and video physics, token scaling, and audiovisual synchronization.

  • Image Generation: Native image generation replaces prompt-based diffusion pipelines with joint text-image modeling, targeting high visual fidelity and compositional controllability.Traditional workflows use LLM-generated prompts with standalone diffusion models but struggle with spatial consistency.
  • Image Generation: Shared-latent Transformer-Diffusion frameworks use unified self-attention to align text structure and image spatial layout early in feature fusion.Ming-Flash-Omni-2.0 uses Mask-based Discrete Diffusion, predicting text tokens alongside continuous features that guide image denoising.
  • Audio Generation: Audio generation must balance semantic-prosody alignment, latency control, and reasoning-streaming synergy, especially for full-duplex conversational scenarios.Continuous latent approaches often improve acoustic fidelity and semantic-prosody alignment, while discrete audio tokens enable autoregressive prediction and instruction following.
  • Audio Generation: Thinker-Talker architectures separate long-form background reasoning from low-latency speech delivery to combine sophisticated voice interaction with streaming responsiveness.A high-capacity Thinker reasons while a lightweight Talker, such as OmniVoice, delivers speech.
  • Video Generation: Video generation faces extreme engineering demands, centered on physics understanding, token explosion, and audio-visual alignment.High-resolution and long videos drive quadratic self-attention growth, causing OOM errors and slower generation after the shift from U-Net to DiT.
  • Video Generation: Video systems pursue compression and sparse attention to reduce memory pressure, while unified audiovisual timelines target millisecond-level synchronization of sound and visuals.Wan2.2 uses Wan-VAE with Flow Matching; audiovisual anchoring locks modalities in a shared time dimension.

3.3 M2M Symmetric Modeling

Symmetrical Multi-Modal models jointly understand and generate multiple modalities within one framework. They divide into fully discretized unified architectures and continuous-space, hybrid-loss approaches, each facing distinct representation and training conflicts.

  • 3.3 M2M Symmetric Modeling: Symmetrical Multi-Modal models support understanding and symmetric generation across multiple modalities within the same framework.Architectures divide into Fully Discretized Unified models and approaches using continuous feature spaces, decoupled encoders, and hybrid losses.
  • Fully Discretized Unified: Fully Discretized Unified models map continuous signals into discrete tokens and train them with a unified autoregressive generation objective.Their advantages include extreme simplicity, but discretization introduces lossy compression and modality competition in the shared Softmax layer.
  • Fully Discretized Unified: Competition-driven latency arises when high-density text tokens and sparse visual/audio tokens share one vocabulary and Softmax, potentially causing exploding output norms and gradient divergence.The differing entropy levels make modalities compete for weight during cross-entropy computation on large-scale data.
  • Continuous Feature Spaces: Continuous-space approaches preserve modality-specific traits with decoupled encoders and hybrid losses, but face comprehension-generation interference and the challenge of bridging AR with Diffusion.Understanding favors compressed semantic abstractions, whereas generation requires fine-grained features for reconstruction; Transfusion uses discrete NTP loss for text and continuous denoising Diffusion loss for image patches.

4 Dataset

NMM training uses heterogeneous mixtures spanning text, images, video, audio, documents, GUI states, tool-use traces, and preference signals, with differing input-output structures and supervision granularity. The dataset landscape progresses from multimodal understanding and generation toward active interaction and behavioral alignment.

  • Dataset Landscape: NMM datasets span heterogeneous modalities and supervision structures, including text, images, videos, audio, documents, GUI states, tool-use traces, and preference signals.These sources differ in modality coverage, input-output structure, and supervision granularity.
  • Understanding-Oriented Data: Understanding-oriented data evolved from image-text alignment toward task-specific reasoning, structured document comprehension, fine-grained grounding, and temporal or audio understanding.This shift trains models to inspect multimodal evidence, integrate inputs, and reason over structure and time rather than merely associate images with captions.
  • Generation-Oriented Data: Generation-oriented data progressed from text-conditioned image synthesis to editing, structured and interleaved generation, temporal video generation, and speech or audio generation.Native models are increasingly expected to create, modify, and organize multimodal content.
  • Interaction-Oriented Data: Interaction-oriented data shifts NMM systems from passive modeling to active agency through web navigation, GUI operation, tool use, API invocation, and embodied control.Across these settings, action trajectories—not merely final answers—provide the key supervision signal.
  • Preference & Alignment Data: Preference and alignment data calibrates behavior by improving visual grounding, reducing hallucination, enhancing safety, aligning multimodal generation with human preferences, and supporting reliable tool use.For multimodal understanding, preference training teaches models to answer according to visual evidence actually present in the input.

5 Training

Training strategies are coupled to architecture, with each fusion regime imposing a distinct training signature across pretraining, supervised fine-tuning, and reinforcement learning. The section traces this evolution from late- through mid- to early-fusion and closes with On-Policy Distillation as an emerging post-RL paradigm.

  • 5 Training: Each fusion regime imposes a distinct training signature spanning freezing topology, learning-rate topology, loss formulation, stability prescription, and curriculum scheduling.The curriculum covers resolutions, sequence lengths, and modality mixtures.
  • 5 Training: The training stack evolves with multimodal fusion from late- through mid- to early-fusion across pretraining, supervised fine-tuning, and reinforcement learning.Sections 5.1–5.3 trace this fusion-coupled evolution across the three training stages.
  • 5 Training: On-Policy Distillation is discussed as an emerging post-RL training paradigm.It concludes the training section in Section 5.4.

5.1 Pre-Training (PT)

Pre-training signatures differ sharply by fusion regime: mid-fusion introduces encoder gradients through progressive unfreezing, differential learning rates, decoupled losses, and curricula, while early-fusion trains all modules jointly under unified objectives requiring stabilization and modality-mixture scheduling.

  • All Fusion Regimes: Frozen modal quantizers and VAEs define the latent space across fusion regimes, so changing them during training would invalidate learned representations.This includes discrete visual and audio tokenizers and continuous VAEs.
  • Mid-Fusion: Mid-fusion begins encoder gradient updates, making progressive unfreezing, differential learning rates, and decoupled losses its defining training techniques.The encoder is initially frozen for alignment and later unlocked, while separate understanding and generation losses can share one backbone.
  • Mid-Fusion: Once encoder gradients are enabled, differential rates become mandatory because one learning rate is typically too high for the encoder yet too low for the LLM.CogVLM applies 1/10 of the base rate to its EVA2-CLIP-E encoder, while Janus-Pro decays 10−3 →10−4 →4×10−5 across three stages.
  • Mid-Fusion: Resolution and context-length curricula link which parameters are unfrozen to the resolution or context at which they are trained, avoiding optimization shock and supporting long audio-video inputs.Examples include MiniCPM-V’s 224 →448 →1344+ resolution progression and Qwen2.5-Omni’s 8,192-to-32,768-token context growth.
  • Early-Fusion: Early-fusion trains all modules jointly from the first step, collapsing modalities into a unified next-token-prediction objective over a shared vocabulary.Discrete-token systems expand the vocabulary with modality codebook tokens, while hybrid variants can combine language modeling with diffusion.
  • Early-Fusion: Unified early-fusion softmaxes require stabilization and modality-mixture scheduling: without QK-Norm, Chameleon diverges after approximately 20% of training, while imbalanced mixtures induce degenerate unconditional priors.Z-loss uses 10−5 · log2 Z to keep logits bounded across heterogeneous token distributions.

5.2 Supervised Fine-Tuning (SFT)

SFT preserves the pretraining signature while introducing regime-specific training choices. Mid-fusion uniquely permits freezing-topology rewiring and pathway-specific scheduling, whereas late- and early-fusion remain constrained to their established trainability structures.

  • Mid-fusion: Mid-fusion uniquely permits SFT to rewire the freezing topology through unfreezing previously frozen components or re-freezing previously trained ones.This flexibility yields two strategies: unfreeze-at-SFT and train-then-re-freeze.
  • Mid-fusion: Unfreeze-at-SFT unlocks encoders only during SFT, while Janus-Pro can keep its generation tokenizer frozen and CogVLM uses one-tenth of the base learning rate.CogVLM, Janus-Pro, and MiniCPM-V freeze the encoder during pretraining and unfreeze it during SFT.
  • Mid-fusion: Train-then-re-freeze is exemplified by Qwen2-VL, which re-freezes its ViT during SFT and tunes only the LLM on ChatML conversations.The ViT was trained through both pretraining stages before being re-frozen.
  • Mid-fusion: Mid-fusion SFT independently schedules understanding and generation pathways, including Janus-Pro’s shift from a 50/50 to 40/60 generation-to-understanding data ratio.The shift biases training toward understanding while retaining generation capability.
  • Early-fusion: Early-fusion cannot rewire freezing and instead trains the universal layer with lower learning rates, prompt-token loss masking, and extra dropout, including 0.05 in Chameleon at 34B scale.AnyGPT instead freezes its LLM and updates only multimodal interface layers for 5,000 steps, while BAGEL trains all components jointly.

5.3 Reinforcement Learning (RL)

In native multimodal RL, the fusion regime determines which parameters receive reward-driven updates and is the central design choice. Late-, mid-, and early-fusion regimes therefore differ in optimization scope, tooling, and structurally exposed failure modes.

  • Scope Across Fusion Regimes: The fusion regime, rather than the RL algorithm, determines the trainable parameter scope and is the most consequential design choice for native multimodal models.RL inherits the scope question from SFT but answers it according to which parameters the reward signal touches and at what cost.
  • Regime-Independent Toolkit: DPO, PPO, and GRPO dominate the regime-independent RL toolkit, while online methods trade higher sampling costs for stronger exploration under sparse preference coverage.Reward design includes outcome-level feedback, whose single scalar per response remains vulnerable to hacking.
  • Late-Fusion RL: Late-fusion RL targets an isolated quality-localizable head, with Qwen2.5-Omni and Qwen3-Omni applying DPO only to the Talker while leaving Thinker and encoders untouched.This yields the simplest configuration: offline DPO with rule-based rewards; the thin projector also limits drift from visual conditioning.
  • Mid-Fusion RL: Mid-fusion RL routes gradients only through the optimized pathway, freezing the VAE and text encoder in generators while updating the diffusion transformer.Naive DPO can ignore image conditions and learn text-only preferences; mDPO conditions the preference loss on the image to counteract this failure.
  • Early-Fusion RL: Early-fusion RL necessarily updates the full backbone under a unified softmax, exposing visual-grounding hacking, perceptual-versus-logical supervision problems, and cross-capability regressions.Emu3.5 and UniRL use full-policy GRPO; Fact-RLHF, shortcut-aware MM-RM, visual-faithfulness terms, and multimodal process rewards address these structural risks.

5.4 On-Policy Distillation (OPD)

On-policy distillation (OPD), including its multi-teacher form MOPD, modifies GRPO with stop-gradient teacher supervision while preserving on-policy student sampling. MiMo-V2.5 reports the first public MOPD deployment on a native multimodal model, using it as a terminal consolidation stage for perception, reasoning, and agentic capabilities.

  • Method: OPD replaces GRPO’s group relative advantage with a stop-gradient reverse-KL log-ratio against a teacher.This is described as a single-line modification of GRPO.
  • Method: Each student-sampled token receives dense, per-position teacher supervision while remaining on-policy.The supervision is defined through the teacher-to-student token-probability log-ratio.
  • MOPD on native multimodal models: MiMo-V2.5 provides the first publicly reported deployment of MOPD on a native multimodal model.Its sequence is text PT → projector warmup → multimodal PT → SFT and agentic post-training → RL and MOPD.
  • MOPD on native multimodal models: MOPD serves as MiMo-V2.5’s terminal consolidation step for strengthening perception, reasoning, and agentic capabilities in one shared backbone.The context is progressively extended from 32K to 1M before RL and MOPD.
  • MOPD on native multimodal models: MiMo-V2.5’s MOPD uses specialist teachers, outcome-reward augmentation, and a permissive teacher pool including domain SFT, RL specialists, and a frozen student snapshot.The outcome-reward augmentation decouples the student from any single teacher’s ceiling, while the snapshot anchors against drift on unfamiliar prompts.

6 Inference & Deployment

Native multimodal inference intensifies long-context and computational bottlenecks because visual and temporal inputs expand into massive token sequences. Deployment therefore emphasizes token reduction, efficient routing, streaming and duplex decoding, and resource-adaptive compression to control latency, memory, and responsiveness.

  • Inference Constraints: Native multimodal inputs can expand into millions of visual and temporal tokens, making prefill cost, KV-cache capacity, and memory bandwidth central inference constraints.Inference efficiency depends on more than parameter count because multimodal tokens must share the context window with language tokens.
  • Token-Efficient Perception: Visual resampling and dynamic-resolution perception reduce unnecessary visual tokens while preserving spatial and temporal grounding across variable-resolution inputs.Fixed-budget resamplers stabilize prefill latency, while dynamic tokenization adapts encoding to aspect ratio and information density.
  • Efficient Architectures: Pure discrete tokenization alleviates memory-bandwidth congestion, while MoE routing prunes activations to address computational bottlenecks caused by sequence expansion.Chameleon uses an 8,192-entry independent image codebook, and Kimi2.5 applies routing strategies for low-cost concurrent inference.
  • Streaming Deployment: NMM deployment is shifting toward streaming decoding, duplex concurrency, and resource-adaptive serving, with TTFT, sustained latency, and real-time responsiveness as primary targets.This transition addresses latency and first-token delays from dynamically arriving multimodal streams.
  • Adaptive Serving: Incremental decoding, full-duplex state management, adaptive bitrate control, and modality-aware mixed quantization jointly trade off interaction continuity, fidelity, memory, and latency.These methods support progressive token emission, concurrent input-output processing, runtime bandwidth adaptation, and differentiated precision across model components.

7 Evaluation

Evaluation of native multimodal models must jointly assess understanding and generation across modalities, testing whether deep cross-modal fusion improves both without degrading either. The evaluation landscape therefore spans perception, reasoning, grounding, synthesis, editing, controllability, interaction, and efficiency-aware protocols.

  • Evaluation Framework: Native multimodal evaluation spans understanding and generation, requiring improved performance on both axes without degradation on either.Understanding includes perception, reasoning, and grounding; generation includes synthesis, editing, and controllability.
  • Understanding Benchmarks: Image understanding benchmarks progress from general perception to knowledge-intensive reasoning, hallucination diagnosis, and document comprehension.VQAv2, GQA, SEED-Bench, MMBench, MMStar, and MMMU cover complementary perception and reasoning capabilities.
  • Generation Benchmarks: Generation benchmarks increasingly test semantic composition and multimodal quality beyond distribution-level similarity, including attribute binding, spatial relationships, counting, and dense prompt following.FID measures distributional similarity, while GenEval, DPG-Bench, and T2I-CompBench target compositional and semantic-level assessment.
  • Audio Evaluation: Audio evaluation covers speech recognition, synthesis, and full-duplex conversation using WER, MOS, latency, synchronization, voice cloning, turn-taking, and interruption metrics.Full-duplex protocols assess simultaneous listening and speaking, natural turn-taking, barge-in handling, response latency, and false interruption rate.
  • Video Evaluation: ResAdapt eliminates over 90% of visual tokens while processing 16× more frames and achieves >15% relative gains on complex long-video reasoning.This demonstrates an evaluation dimension that jointly assesses accuracy and computational cost.

8 Future Outlook

Future NMM progress depends on collapsing fragmented M2T, M2G, and M2M regimes into unified architectures, data, training, deployment, and evaluation. The roadmap ultimately points toward born-native world models that perceive sensory streams, maintain persistent state, and act continuously.

  • Architectural convergence: NMM convergence requires unifying understanding and generation under a single backbone, objective, tokenization scheme, or continuous latent grammar without quality regression.Current unified models often combine next-token prediction with diffusion or flow-matching heads.
  • Architectural convergence: Future native models must scale modality-aware experts while extending nativity beyond text, image, audio, and video to embodied and structured signals.Open questions include expert specialization, cross-modal attention, and nativity for heterogeneous or continuous-time inputs.
  • Data and training: Aligned long-horizon multimodal data, interaction-grounded timing annotations, and scalable generative-modality preferences remain central data bottlenecks.Needed signals include synchronized audio, transcripts, actions, reasoning traces, response timing, aesthetics, factuality, and audio-visual synchronization.
  • Data and training: Native training requires modality-balanced optimization and new reinforcement-learning methods that unify policy gradients with diffusion or flow-based generation.Token-budget allocation, per-modality loss weighting, curriculum scheduling, and stepwise multimodal advantage estimation remain underexplored.
  • Systems and deployment: Deployment research must co-design sparse multimodal MoE systems with adaptive perception and born-streaming, full-duplex interaction to achieve efficient, stable, low-latency operation.ResAdapt eliminated > 90% of visual tokens while expanding the temporal horizon 16×, while streaming systems remain an industrial open problem.
  • Evaluation: Benchmarks should jointly evaluate understanding and generation, response timing, token budget, latency, energy, and robustness against multimodal attacks and generated-content hallucination.The proposed directions include symmetric M2M tasks, temporally-aware metrics, efficiency-aware Pareto protocols, and multimodal safety evaluation.
Loading 2605.25343v1…