PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding and Harnessing Sparsity in Unified Multimodal Models

Shwai He, Chaorui Deng, Ang Li, Shen Yan

arXiv:2512.02351v1cs.CVcs.AI

TL;DR

Unified multimodal models can use more capacity than particular tasks or samples require, but how this inefficiency varies across components is not well understood. The paper probes component compressibility and activation patterns, then adapts generation with sparse experts. Understanding components are compressible, while generation components are more compression-sensitive; adapted BAGEL matches full-model performance while activating about half its neurons.

  • Problem

    How inefficiency varies across unified multimodal model components remains insufficiently understood, although some tasks or samples may not require the model’s full capacity.

    Source: §1 · p. 1

  • Method

    The paper probes component importance with training-free pruning and adapts generation using expert-partitioned MLPs with sparse activation.

    Source: §1 · pp. 1–2

  • Results

    Understanding components are compressible across understanding and generation tasks, whereas generation components are compression-sensitive; adapted BAGEL matches full-model performance while activating about half its neurons.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The findings distinguish component-specific compression behavior and support dynamic activation of generation parameters under sparse inference.

    Source: §5.4 · p. 7, §6 · p. 9

Abstract

from arXiv · show

Large multimodal models have achieved remarkable progress in both understanding and generation. Recent efforts pursue unified multimodal models that integrate heterogeneous components to support both capabilities within a single framework. However, such unification introduces inference inefficiencies, e.g., specific tasks or samples may not require the full knowledge or capacity of the unified model. Yet, a systematic understanding of how these inefficiencies manifest across different components remains limited. In this work, we first conduct a systematic analysis of unified multimodal model components using training-free pruning as a probing methodology, considering both depth pruning and width reduction. Our study reveals that the understanding component exhibits notable compressibility in both understanding and generation tasks, which is more pronounced in the latter. In contrast, the generation components are highly sensitive to compression, with performance deteriorating sharply even under moderate compression ratios. To address this limitation, we propose the Mixture-of-Experts (MoE) Adaptation, inspired by the dynamic activation patterns observed across different samples. This approach partitions the generation module into multiple experts and enables sparse activation to restore generation quality. We validate the effectiveness of sparse activation through expert-frozen tuning and further demonstrate that a fully trainable adaptation delivers additional gains. As a result, the adapted BAGEL model achieves performance comparable to the full model while activating only about half of its parameters. The code is released at \href{https://github.com/Shwai-He/SparseUnifiedModel}{this link}.

1. Introduction

Unified multimodal models combine understanding and generation, but their shared capacity can be inefficient when tasks or samples need only part of the model. The paper analyzes component compressibility and activation patterns, then proposes sparse expert adaptation for generation.

1. Introduction

  • Tasks and samples may rely on only a slimmer or sparsely activated part of a unified model, while component roles, task-specific activation, and input variability create distinct efficiency challenges.The introduction identifies component-wise redundancy, different parameter subsets activated by different tasks, and variation among inputs within a task.Source: §1 · p. 1
  • Training-free pruning probes component importance by removing structures and observing performance changes without retraining.The analysis examines understanding components shared across modalities and their role in producing representations that guide generation.Source: §1 · p. 1
  • Understanding components are more compressible in generation tasks than in understanding tasks, and the two task types predominantly activate different model partitions.These activation patterns motivate dynamic activation for different testing tasks.Source: §1 · p. 2
  • MoE Adaptation partitions generation-module MLP neurons into experts for selective activation, combining expert-frozen tuning with fully end-to-end training.The adapted BAGEL model matches full-model performance while activating about half of its neurons.Source: §1 · p. 2

2. Related Works

Prior work developed separate paradigms for multimodal understanding and generation, while newer systems unify them. This unification raises efficiency questions that existing compression work has not systematically addressed.

2. Related Works

  • Earlier multimodal research treated understanding and generation separately, while recent unified models combine heterogeneous components to support both within one framework.Examples include BAGEL’s mixture-of-transformers design and Ming-Omni’s mixture-of-experts backbone with modality-specific decoders.Source: §2 · p. 2
  • Unified models’ architectural complexity poses efficiency challenges that remain underexplored.The passage presents this challenge in the context of BAGEL and Ming-Omni’s versatile architectures.Source: §2 · p. 2
  • Pruning and width compression target redundancy in neural networks, but whether unimodal compression methods transfer to unified models remains unclear.Prior work on language models and vision-language models motivates systematic study of heterogeneous components with distinct roles.Source: §2 · p. 2

3. Preliminaries: Unified Multimodal Models

Unified multimodal models handle understanding and generation within one framework. Their understanding component produces language representations from multimodal inputs, and the generation component uses those representations to synthesize non-text outputs.

3. Preliminaries: Unified Multimodal Models

  • Unified multimodal models integrate multimodal understanding and generation within a single framework.Source: §3 · p. 2
  • For understanding, the model predicts a sequence of textual output tokens autoregressively from a multimodal input.The passage identifies θund as the understanding-component parameters and yund as the output-token sequence.Source: §3 · p. 2
  • For generation, the understanding component processes an instructional input into conditional features that the generative component uses, alongside a generative input, to synthesize an output.Understanding handles inputs from all modalities, whereas generation operates on non-text outputs such as images or audio; their different objectives and pathways may yield different compression behavior.Source: §3 · p. 3

4. Methodology

The methodology uses training-free pruning to probe architectural redundancy, then applies training-aware MoE adaptation to maintain performance under sparse activation. It covers depth pruning, neuron-level width reduction, and dynamic expert routing.

4.1. Training-free Compression Strategies

  • Depth pruning measures layer redundancy through input-output cosine similarity, with higher similarity indicating a more marginal transformation.The method extends this metric, previously used in unimodal LLMs, to unified models.Source: §4.1 · p. 3
  • Neuron partition reduces MLP width by retaining important hidden neurons and pruning less important ones, using activation magnitude and output amplification to score importance.The approach removes whole neurons rather than individual weights, supporting hardware-friendly efficiency.Source: §4.1 · p. 3
  • Understanding and generation tasks have relatively low overlap among their respective top-50% important neurons, motivating calibration samples aligned to the target task.The method partitions neurons into understanding-only, generation-only, and shared groups.Source: §4.1 · p. 3

4.2. Training-Aware MoE Adaptation

  • Generation neurons show sample-dependent activation across prompts and timesteps, motivating dynamic rather than static parameter selection.Only a small subset remains consistently active throughout inference.Source: §4.2 · p. 4
  • Expert partition preserves neurons important across calibration samples as shared experts and allocates the remaining, more sample-dependent neurons to routed experts.Importance-ranked neurons are assigned in alternating forward and reverse order to balance total importance across experts.Source: §4.2 · p. 4
  • MoE adaptation inserts a per-layer router to select routed experts, first tuning the router and other parameters with experts frozen, then releasing the freeze.The gating function and shared and routed expert transformations define the MoE layer output.Source: §4.2 · pp. 4–5

5. Experiments

Experiments show that understanding components tolerate more compression than generation components, while sparse MoE adaptation can recover generation quality. Calibration alignment and compression type matter, and depth pruning can sharply harm understanding-task performance.

5.1. Experimental Setup

  • Experiments cover BAGEL, Ming-Omni, and Qwen-Image; MoE adaptation compares generation-only partitioning with joint partitioning that keeps understanding experts frozen.Training-free compression uses target-task calibration examples; MoE adaptation also uses image–text pairs and synthetic data.Source: §5.1 · p. 5, §5.4 · pp. 7–8

5.2. Redundancy of Und. Component

  • Removing half the MLP layers drops MME perception from 1684.8 to 304.5 and cognition from 696.7 to 127.1.The paper links this collapse to autoregressive error accumulation in language-response tasks.Source: §5.2 · p. 5
  • At 50% depth reduction, understanding-component pruning is effective for BAGEL and Qwen-Image on GenEval, but less effective for Ming-Omni.The authors attribute Ming-Omni’s lower tolerance to its smaller generation component’s greater dependence on precise understanding features.Source: Figure 4 · p. 5, §5.2 · p. 5
  • Neuron partition consistently outperforms layer dropping on understanding tasks, while retaining important neurons supports more reliable compression.Compression should be more conservative for understanding tasks than for generation tasks.Source: §5.2 · pp. 5–6, Table 2 · p. 5
  • MMBench scores are 79.2 with understanding-task calibration and 74.8 with generation-task calibration; generation-calibrated outputs also better match prompts.The figure compares unmodified outputs with partitioned outputs calibrated on image-generation or understanding data.Source: §5.2 · pp. 6–7, Figure 5 · p. 6

5.3. Dilemma of Gen. Component Compression

  • A 50% width reduction in the generation component substantially degrades output fidelity and coherence, with distorted structures and unrealistic textures.The qualitative comparison uses the prompt “The word START.”Source: §5.3 · p. 7, Figure 6 · p. 7

5.4. MoE Adaptation for Gen. Component Sparsity

  • With Expert-Frozen Tuning, GenEval improves from 0.58 to 0.78, reflecting more coherent, visually faithful outputs and stronger alignment between generated content and the given instructions.Source: §5.4 · p. 7
  • Full MoE Adaptation further improves generation quality after Expert-Frozen Tuning across the Gen. and Und. & Gen. configurations.Table 4 reports activated parameter counts for both understanding and generation components.Source: §5.4 · p. 8, Table 4 · p. 8

6. Conclusion

The conclusion contrasts the compressibility of understanding components with the sensitivity of generation components, then presents MoE Adaptation as a way to restore generation quality under sparse activation.

6. Conclusion

  • Understanding components are strongly compressible across understanding and generation tasks, while generation components can suffer catastrophic quality loss under moderate pruning.The paper reports that dynamic MoE activation effectively restores generation quality under sparse activation.Source: §6 · p. 9
  • MoE Adaptation dynamically activates generation parameters, drawing on activation patterns that vary across samples.The study combines training-free analysis with training-aware adaptation to examine parameter efficiency.Source: §6 · p. 9
  • The paper identifies substantial optimization space for parameter efficiency in unified multimodal modeling through training-free analysis and training-aware adaptation.Source: §6 · p. 9

A. Generation Component Compression

Static compression of generation components is fragile across both width and depth, and attention compression is also sensitive to even modest reductions.

A. Generation Component Compression

B. Attention Head Partition

The paper focuses width reduction on MLP layers and extends its compression metric to attention heads.

B. Attention Head Partition

C. Depth Reduction on Understanding Tasks

Depth reduction has limited impact on generation but severely harms multimodal understanding. In autoregressive decoding, early deviations compound until answers collapse.

D. Comparison with Compression Methods

Neuron Partition compares favorably with gradient-based pruning and 4-bit quantization for the understanding component. It also avoids labeled data and explicit gradient computation.

E. Calibration Data for Neuron Partitioning

Calibration data affects activation-score measurement and neuron selection. The section includes calibration-dataset ablations for understanding and generation, alongside an example Neuron Partition output.

Loading 2512.02351v1…