Source-linked AI summary
DecoMoE: Decoupling Visual Propagation and Expert Computation for Efficient Multimodal MoE Inference
Xudong Tan, Peng Ye, Ming Xie, Chenyu Huang, Yaoxin Yang, Jiayuan Fan, Tao Chen
TL;DR
Long visual-token sequences make multimodal MoE inference costly, and compressing only tokens or experts leaves redundancy along the other dimension. DecoMoE combines sample-adaptive visual-token removal with text-routing-calibrated contiguous expert prefixes. On Qwen3-VL-MoE, it retains 97.91% of baseline performance and achieves a 1.69× speedup.
Problem
Long visual-token sequences repeatedly incur attention, routing, dispatch, and expert-MLP computation, while compressing only the token or expert dimension leaves redundancy in the other.
Source: §1 · p. 1Method
DecoMoE decouples visual propagation from expert computation using a sample-adaptive visual boundary and text-routing-calibrated contiguous expert prefixes.
Source: §1 · p. 3Results
On Qwen3-VL-MoE, DecoMoE retains 97.91% of baseline performance while reducing computation from 27.06 to 16.73 TFLOPs and latency from 0.44 to 0.26 seconds, for a 1.69× speedup.
Source: §4.2 · p. 7Takeaways & Limitations
The two mechanisms provide structured reductions along the token and expert dimensions, with a 1.69× measured speedup and 2.09% average performance reduction on Qwen3-VL-MoE relative to the dense baseline.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Multimodal mixture-of-experts (MoE) models combine sparse expert activation with visual-language capabilities, yet their inference remains costly because long visual-token sequences repeatedly incur attention, routing, dispatch, and expert-MLP computation. Existing methods typically compress either the token or expert dimension, leaving redundancy along the other. Our analysis reveals two complementary regularities: the depth required for visual propagation varies across inputs, while text-token routing exhibits concentrated and recurrent expert-importance patterns. Based on these observations, we propose DecoMoE, a two-dimensional structured compression framework that decouples visual propagation from expert computation. The Sample-Adaptive Visual Boundary (SAVB) predicts an input-dependent visual-exit layer at which the visual-token block is removed. The Routing-Calibrated Expert Prefix (RCEP) reorders experts offline using text-token routed mass and, from this predicted exit layer onward, retains at each MoE layer the shortest contiguous prefix covering a target routed-mass fraction. We evaluate DecoMoE on Qwen3-VL-MoE and InternVL3.5-30B-A3B across six benchmarks. On Qwen3-VL-MoE, DecoMoE retains 97.91% of dense-baseline performance while reducing computation from 27.06 to 16.73 TFLOPs and latency from 0.44 to 0.26 seconds, yielding a 1.69x speedup. Code will be available at https://github.com/ShawnTan86/DecoMoE.
1 INTRODUCTION
Multimodal MoE inference remains costly because visual tokens incur repeated computation, and compressing only tokens or experts leaves redundancy in the other dimension. DecoMoE combines an input-dependent visual boundary with text-routing-calibrated contiguous expert prefixes; on Qwen3-VL-MoE, it achieves a 1.69× speedup with a 2.09% average performance reduction relative to the dense baseline.
1 INTRODUCTION
- Long visual-token sequences repeatedly incur attention, routing, dispatch, and expert-MLP computation, while compressing only tokens or experts leaves redundancy along the other dimension.The token dimension determines sequence length and attention cost; the expert dimension determines expert span, dispatch, and MLP costs.Source: §1 · p. 1
- Visual-exit depth varies across inputs, while text-token routing has more concentrated routed mass and greater cross-case expert overlap than visual-token routing.These regularities motivate sample-level control of visual propagation and text-aligned calibration of expert importance.Source: §1 · p. 2
- SAVB removes the visual-token block at a sample-dependent boundary, and RCEP retains a contiguous expert prefix calibrated from text-token routed mass.RCEP activates from the triggering layer onward and selects the shortest prefix covering a target fraction of current activated routed mass.Source: §1 · pp. 2–3
- 1.69× measured speedup with a 2.09% average performance reduction relative to the dense baseline on Qwen3-VL-MoE.DecoMoE is evaluated on Qwen3-VL-MoE and InternVL3.5-30B-A3B across six benchmarks.Source: §1 · p. 3
2 RELATED WORK
Multimodal MoE models combine visual understanding with sparse token-wise expert activation, but their inference costs depend jointly on sequence length and routed expert computation. Existing acceleration methods often target one dimension or adapt computation without organizing it as DecoMoE does: an input-dependent visual boundary followed by text-calibrated contiguous expert prefixes.
2.1 MULTIMODAL MIXTURE-OF-EXPERTS MODELS
- Multimodal MoE models scale capacity through token-wise conditional routing, but long visual sequences still incur attention, routing, dispatch, and expert-MLP costs at retained layers.Inference cost jointly depends on sequence length and routed expert computation, and visual and textual tokens may induce different expert loads.Source: §2.1 · p. 3
2.2 EFFICIENT MULTIMODAL MIXTURE-OF-EXPERTS MODELS
- Visual-token compression and expert-computation methods address complementary dimensions, while DecoMoE uses an input-dependent visual boundary and cross-case text-routing regularity to retain contiguous expert prefixes.Figure 2 outlines SAVB's exit decision and RCEP's offline expert ordering and online mass-sufficient prefix selection.Source: §2.2 · p. 3, Figure 2 · p. 4
3 METHODOLOGY
DecoMoE decouples visual-propagation depth from the expert structure used for subsequent text-only computation. It uses an input-dependent visual-exit boundary and a text-routing-calibrated expert prefix to structure compression along both dimensions.
3 METHODOLOGY
- DecoMoE uses SAVB to determine when computation becomes text-only, then RCEP selects a contiguous prefix of reordered experts at each subsequent MoE layer.The visual boundary and layer-wise prefix lengths remain input-dependent, while execution is structurally regular along the token and expert dimensions.Source: §3 · p. 4
3.1 PRINCIPLE
- Visual-exit layers preserving answer correctness vary across inputs, while text-token routing shows greater routed-mass concentration and cross-case expert overlap than visual-token routing.Figure 3(a) depicts forced-exit perplexity and earliest correct exits; panels (b) and (c) compare routed-mass concentration and repeated experts.Source: §3.1 · p. 4, Figure 3 · p. 5
3.2 SAMPLE-ADAPTIVE VISUAL-PROPAGATION DECOUPLING
- SAVB evaluates a lightweight gate at candidate layers spaced four decoder layers apart and removes the complete visual-token block before self-attention when it triggers.Gate labels combine forced-exit correctness with a percentile-based cutoff on the reference answer’s first-token NLL; the backbone remains frozen during training.Source: §3.2 · p. 5
3.3 TEXT-ALIGNED EXPERT-IMPORTANCE DECOUPLING WITH ROUTING-CALIBRATED PREFIXES
- RCEP orders experts offline by mean text-token routed mass, then dynamically selects a layer-wise prefix from the visual-exit layer onward to cover a target routed-mass fraction.Within the prefix, routing is recomputed and renormalized; each token remains dispatched to k_moe experts, while experts outside the prefix are excluded from routing and MLP evaluation.Source: §3.3 · p. 6
4 EXPERIMENTS
Across two multimodal MoE backbones and six benchmarks, DecoMoE combines input-adaptive visual exits with routing-calibrated expert prefixes. The experiments report high baseline retention alongside lower computation and latency, and analyze the behavior of both components.
4.1 EXPERIMENTAL SETUP
- The evaluation covers two backbones and six accuracy-like benchmarks; POPE and MME calibration data are excluded from the reported benchmarks.SAVB is supervised on a random 50% subset and RCEP calibrated on a random 10% subset of POPE and MME; the backbone remains frozen.Source: §4.1 · p. 7
4.2 MAIN RESULTS
- 97.91% baseline retention with computation falling from 27.06 to 16.73 TFLOPs and latency from 0.44 to 0.26 seconds on Qwen3-VL-MoE yields a 1.69× speedup.InternVL3.5-30B-A3B retains 97.87% with computation falling from 9.86 to 5.97 TFLOPs and latency from 1.50 to 1.06 seconds; FastV and SparseVLM use fewer TFLOPs but exceed dense latency on Qwen3-VL-MoE.Source: §4.2 · pp. 7–8
4.3 COMPONENT-WISE ABLATION STUDY
- 97.41% retention with SAVB exceeds 96.18% with fixed L28; dynamic RCEP averages 35.01 experts and reaches 97.83%, versus 97.48% for a fixed 50-expert prefix.Full DecoMoE also achieves the lowest latency at 0.69× baseline, supporting complementary reductions from sample-adaptive visual exits and mass-adaptive expert selection.Source: Table 2 · p. 8, §4.3 · p. 8
4.4.1 DATASET-WISE VISUAL-EXIT BEHAVIOR
- 77.3% of MMBench cases exit at layers 18–27, while 72.2% of HallusionBench cases exit at layers 28–37; 34.2% of ScienceQA cases do not trigger an exit.POPE exits are distributed across multiple intervals, consistent with dataset-dependent visual-exit decisions rather than a single global boundary.Source: §4.4.1 · p. 8
4.4.2 VISUAL-PROPAGATION DECOUPLING VALIDATION
- 1.1060 PPL at the predicted boundary is close to 1.1105 with no exit; exiting at offset −8 reaches 1.2666, while later offsets remain at 1.0983–1.1038.The figure tests forced visual exits from eight layers before to eight layers after the predicted boundary.Source: §4.4.2 · p. 8, Figure 5 · p. 9
4.6 STRUCTURED EXECUTION FOR REALIZED ACCELERATION
- 97.05% retention at 0.29 seconds on AI2D and HallusionBench; versus the dense baseline, TTFT falls 30.8% from 370.80 to 256.53 ms and TPOT falls 30.8% from 135.80 to 93.95 ms/token.SAVB removes a contiguous visual-token block and RCEP limits computation to a contiguous expert prefix, reducing irregular selection and fragmented dispatch.Source: Figure 5 · p. 9, Table 4 · p. 9, §4.6 · p. 9
4.5.1 MASS–PERFORMANCE VALIDATION OF EXPERT REORDERING
- At a 10% keep ratio, RCEP retains 96.9% of baseline, versus 87.3% for Native and 85.4% for Random; across five budgets, the averages are 97.5%, 93.9%, and 92.3%, respectively.The analysis finds that Mean-Mass better identifies experts important for text-token computation and organizes them into a compact prefix.Source: §4.5.1 · p. 9
4.5.2 TOP-p SENSITIVITY AND DYNAMIC PREFIX SELECTION
- At p = 0.70, RCEP reaches 97.8% retention with an average prefix of 35.0 experts; layer-varying routed-mass concentration and overlap can make a fixed prefix size suboptimal.Retention changes little once the keep ratio reaches 30%, indicating that a fixed large expert budget may be unnecessary.Source: §4.5.2 · p. 9
5 CONCLUSION
DecoMoE separates input-dependent visual propagation from text-token expert importance, representing the latter as a compact contiguous prefix. On Qwen3-VL-MoE, it achieves a 1.69× measured speedup with a 2.09% average performance reduction relative to the dense baseline.
5 CONCLUSION
- The conclusion presents DecoMoE as a two-dimensional structured-compression framework that decouples visual propagation from expert computation.Visual exits are input-dependent; expert importance for subsequent text computation is organized into a compact contiguous prefix.Source: §5 · p. 10
A APPENDIX
The appendix documents implementation details and additional evidence for SAVB and RCEP, including bottleneck profiling, algorithms, FLOPs accounting, calibration, and routing diagnostics.
A APPENDIX
- The appendix provides implementation-level details and additional evidence for SAVB and RCEP, covering profiling, algorithms, FLOPs accounting, calibration, routing diagnostics, and case studies.It presents the offline and online algorithms and reproduces the analytical FLOPs accounting used in the experiments.Source: A APPENDIX · p. 12
A.1 INFERENCE BOTTLENECK ANALYSIS
Attention and expert MLPs dominate measured decoder-layer latency, while long visual prefixes repeatedly add quadratic attention work and MoE computation. The bottleneck analysis motivates reducing both sequence length and expert work.
A.1 INFERENCE BOTTLENECK ANALYSIS
- Attention typically contributes approximately 57–68% of measured time per layer, and expert MLPs contribute approximately 18–26%.Together they generally exceed 80% after excluding startup and boundary effects.Source: §A.1 · p. 12
- For a prefill sequence of length S, self-attention has an O(S^2) interaction term, so retaining visual tokens across decoder layers repeatedly incurs quadratic attention cost.The same visual tokens also enter each MoE block.Source: §A.1 · p. 12
- Visual-token routing mass is spread across a large fraction of the expert pool, requiring the dense batched expert implementation to maintain a large effective expert set.The passage attributes this observation to Appendix A.6.Source: §A.1 · p. 12
- SAVB shortens the sequence after visual information transfers to text states, and RCEP reduces the physically evaluated expert prefix for the remaining text-only computation.These interventions address the attention-side and expert-side costs, respectively.Source: §A.1 · p. 12
A.2.1 OFFLINE CALIBRATION
Offline calibration keeps the backbone frozen while calibrating SAVB and RCEP separately. The resulting expert permutation is applied consistently to router and expert tensors, changing storage order rather than the unsliced model's function.
A.2.1 OFFLINE CALIBRATION
- The backbone remains frozen throughout offline calibration.SAVB labels come from forced visual exits at candidate layers G = {3, 7, 11, . . . , 47}, using zero-based implementation indices.Source: §A.2.1 · p. 13
- RCEP is calibrated separately from text-token router statistics collected after visual exit.The calibration produces layer-wise expert permutations.Source: §A.2.1 · p. 13
- Each permutation is applied consistently to router output rows and corresponding expert tensors, so reordering changes physical storage but not the unsliced model's function.The corresponding tensors are the gate/up and down expert tensors.Source: §A.2.1 · p. 13
A.2.2 ONLINE INFERENCE AND PREFIX-RESTRICTED ROUTING
After visual exit, RCEP uses full-pool router logits to determine a prefix length, then routes within that prefix. The layer-index convention distinguishes the trigger layer from the first layer evaluated without visual tokens.
A.2.2 ONLINE INFERENCE AND PREFIX-RESTRICTED ROUTING
- RCEP uses full logits over all reordered experts only to estimate the required prefix length, then slices logits, recomputes softmax, reruns native Top-k within the prefix, and renormalizes selected weights.It does not intersect the full-pool Top-k set with the prefix; because K_l ≥ k_moe, prefix routing returns exactly k_moe experts per token.Source: §A.2.2 · p. 13
- A trigger recorded at layer l means that layer l is evaluated with the full multimodal sequence; visual tokens are removed before layer l + 1.Thus a trigger at layer 27 preserves visual tokens through the 28th decoder block.Source: §A.2.2 · p. 13
A.3 ANALYTICAL FLOPS ACCOUNTING
The FLOPs accounting replays decoder costs across matched cases, tracking visual and text tokens, retained expert prefixes, and recurrent decoding. It counts executed expert matrix multiplications while excluding several model and runtime costs.
A.3 ANALYTICAL FLOPS ACCOUNTING
- The FLOPs replay counts decoder-side linear projections, routing, retained-expert costs, and the explicit SAVB MLP overhead.A multiply-add counts as two FLOPs; the SAVB MLP overhead is counted according to the number of gate evaluations.Source: §A.3 · pp. 13–14
- After visual exit, online inference finds the shortest prefix covering target routed mass, enforces K_l ≥ k_moe, and evaluates only the first K_l contiguous expert MLPs.The algorithm applies prefix-restricted routing during autoregressive decoding as well.Source: §A.3 · p. 14
- The Qwen3-VL-MoE-30B-A3B replay uses L = 48, E = 128, k_moe = 8, d = 2048, q = 4096, kv = 512, and m = 768.For each case, recurrent decode forwards are D_c = max(G_c − 1, 0), because prefill produces the first generated token.Source: §A.3 · p. 14
- The expert-cost convention uses K = E = 128 for the baseline and the recorded prefix length K_l for RCEP, reflecting executed dense batched expert matrix multiplications.It does not count only the eight nonzero routing weights.Source: §A.3 · p. 15
- The accounting covers the language decoder backbone and explicit SAVB MLP but excludes the vision encoder, several model operations, and runtime overheads.Theoretical FLOPs and measured profiler CUDA time are complementary quantities, and neither is inferred from the other.Source: §A.3 · p. 15
A.4 CALIBRATION DATA AND DATA ISOLATION
The appendix specifies how SAVB and RCEP are calibrated and reports routing diagnostics, sensitivity results, and cases showing why visual-exit depth varies by task. It also distinguishes strict cross-dataset evaluation from POPE/MME results that include calibration IDs.
A.4.1 SAVB SUPERVISION; A.4.2 RCEP CALIBRATION AND TEXT-ROUTING ALIGNMENT
- SAVB uses forced-exit labels on POPE/MME with frozen features, while RCEP calibrates text-token routed mass and applies online prefix slicing only after SAVB triggers.RCEP collects text-routing statistics after visual-token removal at layer 24, reorders all calibrated MoE layers, and does not constrain the online exit layer.Source: §A.4.1 · p. 15, §A.4.2 · p. 15
A.4.3 LEAKAGE BOUNDARY AND REPORTING PROTOCOL
- POPE/MME aggregates that include calibration IDs are transductive, whereas the six datasets not used for calibration are strict cross-dataset generalization tests.For fully inductive POPE/MME reporting, persist calibration IDs before training and score only complementary IDs.Source: §A.4.3 · p. 15
A.5.1 COMPLETE EXPERT-ORDER COMPARISON
- At a 10% keep ratio, RCEP retains 96.87% of baseline performance, versus 87.26% for Native and 85.43% for Random; mean retentions across seven budgets are 97.47%, 93.02%, and 92.56%, respectively.RCEP achieves the highest retention at every tested budget.Source: Table 5 · p. 16, §A.5.1 · p. 16
A.5.2 COMPLETE ROUTED-MASS TARGET SENSITIVITY
- At p = 0.70, RCEP achieves 97.8% measured retention with an average prefix of 35.0 experts, while prefix length grows monotonically as the mass target increases.Retention remains relatively stable across the p = 0.1 to 1.0 sweep.Source: Table 6 · p. 16, §A.5.2 · p. 16
A.6 ADDITIONAL ROUTED-MASS EVIDENCE
- After visual-token eviction, text-routing maps remain aligned; at 50% routed mass, text tokens generally need 50–80 experts versus 120–128 for visual tokens, with deep-layer cosine similarity above 0.997 and MAE below 0.0003.These diagnostics support applying expert-prefix compression after the visual exit rather than while visual routing remains diffuse.Source: §A.6 · pp. 16–17, Figure 7 · p. 18, Figure 8 · p. 19
A.7.1 CASE 1: MANY VISUAL TOKENS BUT AN EARLY BOUNDARY
- Despite approximately 4,000 visual tokens, removing the visual block after layer 26 preserves fine visual details and the model’s textual response ability.The snowy residential-scene case demonstrates that a long image prefix can exit early once its information is consolidated into text states.Source: §9 · p. 18, Figure 9 · p. 20
A.7.2 CASE 2: FEW VISUAL TOKENS BUT A LATE BOUNDARY
- For an OCR image with approximately 80 visual tokens, exiting at layer 41 causes a recognition and translation failure, while exit at layer 42 preserves the correct phrase and translation.The case shows that token count alone is insufficient to determine the needed exit depth.Source: §A.7.2 · p. 19, Figure 10 · p. 20
A.7.3 CASE 3: THE SAME IMAGE REQUIRES DIFFERENT DEPTHS BY TASK
- With identical visual evidence for all three fire-hydrant prompts, object presence is available early, grounded color justification needs additional propagation, and fine-grained attribute extraction requires a much later boundary.Source: Figure 11 · p. 21, §A.7.3 · p. 21