PaperlayerView exact revision on arXiv

Source-linked AI summary

MMR-Bench: A Comprehensive Benchmark for Multimodal LLM Routing

Haoxuan Ma, Guannan Lai, Han-Jia Ye

arXiv:2601.17814v1cs.AI

TL;DR

MLLM heterogeneity makes fixed-model deployment inefficient because workloads range from lightweight OCR to complex multimodal reasoning, while multimodal routing lacks standardized budget-aware evaluation. MMR-Bench provides a controlled benchmark for routing across candidate MLLMs and shows that multimodal cues improve cost–accuracy trade-offs, with policies also generalizing beyond their training settings.

  • Problem

    No single MLLM is uniformly superior across tasks, and extending query-level routing to multimodal models requires evaluation under heterogeneous modalities, costs, and compute budgets.

  • Method

    MMR-Bench evaluates routing with modality-aware inputs, adjustable compute budgets, standardized costs, diverse vision-language tasks, and single-model, oracle, and routing baselines.

  • Results

    Multimodal signals improve cost–accuracy trade-offs, sometimes matching or exceeding the strongest single model at roughly 33% of its cost, while trained policies generalize without retuning.

  • Takeaways & Limitations

    MMR-Bench supports adaptive multimodal model selection as a practical approach to efficient MLLM deployment under realistic budget constraints.

  • Takeaways & Limitations

    Evaluation is offline: routers use precomputed utilities and costs, cannot access evaluation labels, and issue no multiple adaptive model calls per instance.

Abstract

from arXiv · show

Multimodal large language models (MLLMs) have advanced rapidly, yet heterogeneity in architecture, alignment strategies, and efficiency means that no single model is uniformly superior across tasks. In practical deployments, workloads span lightweight OCR to complex multimodal reasoning; using one MLLM for all queries either over-provisions compute on easy instances or sacrifices accuracy on hard ones. Query-level model selection (routing) addresses this tension, but extending routing from text-only LLMs to MLLMs is nontrivial due to modality fusion, wide variation in computational cost across models, and the absence of a standardized, budget-aware evaluation. We present MMR-Bench, a unified benchmark that isolates the multimodal routing problem and enables comparison under fixed candidate sets and cost models. MMR-Bench provides (i) a controlled environment with modality-aware inputs and variable compute budgets, (ii) a broad suite of vision-language tasks covering OCR, general VQA, and multimodal math reasoning, and (iii) strong single-model reference, oracle upper bounds, and representative routing policies. Using MMR-Bench, we show that incorporating multimodal signals improves routing quality. Empirically, these cues improve the cost-accuracy frontier and enable the routed system to exceed the strongest single model's accuracy at roughly 33% of its cost. Furthermore, policies trained on a subset of models and tasks generalize zero-shot to new datasets and text-only benchmarks without retuning, establishing MMR-Bench as a foundation for studying adaptive multimodal model selection and efficient MLLM deployment. The code will be available at: https://github.com/Hunter-Wrynn/MMR-Bench.

1. Introduction

MMR-Bench addresses the challenge of selecting among heterogeneous MLLMs when task difficulty and compute needs vary. It evaluates whether multimodal routing improves budget-aware accuracy and generalizes across models, tasks, and modalities.

  • Motivation: No single MLLM is uniformly superior across lightweight OCR and complex multimodal reasoning workloads.Using one model for every query can over-allocate compute on easy instances or sacrifice accuracy on difficult ones.
  • Motivation: Text-only routing can outperform some fixed models at equal cost but remains weaker than stronger models, motivating multimodal signals.The limitation is especially relevant when visual evidence determines task difficulty.
  • Benchmark: MMR-Bench provides standardized cost models, adjustable budgets, modality-aware inputs, diverse vision-language tasks, and reference routing baselines.The framework is designed for controlled and reproducible comparisons of adaptive model selection.
  • Findings: Multimodal routing can match or surpass the strongest single model at roughly 33% of its cost.The result is reported as a strictly better cost–accuracy trade-off in certain scenarios.
  • Findings: Routing policies trained on subsets of MLLMs and tasks generalize to unseen data without additional tuning and transfer to text-only routing.This establishes robustness beyond the training distributions described in the benchmark.

2. Related work

Prior work improves efficiency within individual multimodal model families, but MMR-Bench targets selection across a heterogeneous MLLM ecosystem under explicit budgets.

  • MLLM ecosystem: MLLMs differ substantially in architecture, training data, modality alignment, and computational efficiency.This heterogeneous ecosystem includes models with distinct strengths, limitations, and resource requirements.
  • Routing: Text-only routing reduces cost by dispatching queries among heterogeneous LLMs, but multimodal routing must handle cross-modal signals and architecture-level heterogeneity.These factors make confidence estimation and cost modeling more complex than in text-only settings.
  • Research gap: Existing multimodal efficiency methods improve routing within a single model family rather than selecting among the broader MLLM zoo.MMR-Bench addresses this gap by learning which MLLM to use under a given budget.

3. Preliminaries

The paper formulates routing as budget-aware selection among candidate MLLMs using available modalities and observable features. Experiments compare unimodal and multimodal signals through cost–accuracy frontiers.

  • 3.1. Problem definition: MLLM routing selects one candidate model per input to trade off model utility against cost.The setting focuses on text and image inputs, while allowing missing modalities through an availability vector.
  • 3.1. Problem definition: A modality availability vector distinguishes text-only samples from paired text–image inputs.For example, m_i = (1, 0) denotes text-only input, whereas m_i = (1, 1) denotes paired text–image input.
  • 3.1. Problem definition: Routers use text-only, image-only, or multimodal features, with the available modalities determining which information enters the routing decision.Comparing these router families isolates the contribution of each modality.
  • 3.2. Motivation: Cluster-based routing maps each test sample to its nearest cluster and assigns the model selected for that cluster.Varying the cluster count and cost weight produces policies across the cost–accuracy spectrum.
  • 3.2. Motivation: Text-only routing helps at comparable cost but underperforms on text-in-image, dense OCR, charts, spatial reasoning, and low-resolution cases.These errors arise when textual cues provide an unreliable proxy for visual complexity.
  • 3.2. Motivation: Multimodal routers achieve higher accuracy at fixed cost and lower cost at fixed accuracy than text-only and image-only variants.The advantage is stable across cluster and cost-weight ablations and widens when visual clutter or linguistic complexity drives difficulty.

4. MMR-Bench: Benchmark Construction

MMR-Bench is an offline, budget-aware benchmark for evaluating multimodal routing across heterogeneous models, datasets, and scenarios. It combines standardized outcomes, diverse multimodal tasks, and cost-aware evaluation to support reproducible model-selection studies.

  • Design Principles: MMR-Bench provides raw model outputs, task-specific utilities, and normalized inference costs for reproducible offline routing evaluation.The benchmark uses frozen splits and deterministic scoring without rerunning MLLMs.
  • Design Principles: The benchmark targets routing across a heterogeneous MLLM zoo rather than serving as a generic vision-language leaderboard.Its candidate pool spans models with diverse architectures, capacities, and training regimes.
  • Datasets and Model Pool: Its task suite covers document OCR and understanding, general VQA and grounding, and multimodal math and diagram reasoning.These scenarios include OCR, dense layouts, natural images, charts, equations, and diagrams.
  • Datasets and Model Pool: The model pool includes commercial and open-weight MLLMs with heterogeneous architectures, capacities, and cost profiles.Examples include GPT-5, Gemini, Claude, Gemma, InternVL3, and Qwen2.5-VL families.
  • Cost-aware Routing Evaluation Protocol: For each instance–model pair, MMR-Bench records utility and normalized inference cost, then evaluates routers through offline performance–cost curves.Routers may use query features, optional budgets, lightweight benchmark features, and static model metadata, but not evaluation labels or utilities.

5. Experiments

Experiments compare routing methods under fixed splits, shared candidate models, and cost-aware metrics. Matrix-factorization routers perform most reliably in aggregate, while specialized instance-based methods achieve dataset-specific peaks and routing improves the cost–accuracy frontier over fixed models.

  • Experimental Setup: Experiments use frozen 2:8 train/test splits, 10 candidate MLLMs, precomputed utilities and costs, and macroaveraged nAUC and peak score.The shared cost range is defined by always selecting the cheapest versus most expensive single model.
  • Comparative Analysis: LinearMFRouter achieves the highest full-dataset nAUC and peak score, at 0.7042 and 0.7533 respectively.MLPMFRouter is close behind on peak score at 0.7494 and remains competitive on nAUC at 0.6913.
  • Comparative Analysis: Instance-based routers achieve localized wins, with KNNRouter leading peak score on OCRBench and both nAUC and peak score on MathVista.KMeansRouter leads nAUC on OCRBench and MathVision, but these gains are less stable in aggregate.
  • Comparative Analysis: Matrix-factorization routers are consistently superior on general VQA, indicating stronger cross-category robustness than instance-based or clustering heuristics.The analysis attributes their smoother generalization to modeling per-model outcomes in a low-rank space.
  • Overall Results on MMR-Bench: Routing shifts the cost–accuracy frontier upward and leftward relative to fixed models, indicating higher accuracy at equal cost and lower cost at equal accuracy.The x-axis is log-scaled to emphasize the low-cost region.

6. Analyses

The analyses show that adaptive multimodal fusion improves routing, while routers remain robust across datasets and can transfer from multimodal to text-only workloads.

  • Modality gap: Adaptive fusion exposes a modality gap between image and text signals, showing that equal weighting is not uniformly appropriate.It reweights modalities per instance and captures cross-modal agreement or mismatch.
  • Modality gap: +0.3403 ∆nAUC is the largest adaptive-fusion lift, achieved by KMeans, while QNC improves from +∞ to 1.0585.MLP also improves from QNC +∞ to 0.9947, whereas Linear and KNN show smaller or negative changes.
  • Distribution-shift robustness: Cross-dataset routers consistently achieve higher peak score Ps than the best single model and remain close to in-domain routers.The results support robustness to within-scenario distribution shift through transferable modality-aware difficulty cues.
  • Cross-modality transfer: Multimodal routers can transfer across modalities and generalize to unimodal tasks.The reported transfer is attributed to modality-agnostic difficulty cues, fusion-induced regularization, and cost-aware calibration.
  • Cross-modality transfer: A multimodal-trained router retains strong effectiveness on text-only benchmarks, with Ps consistently exceeding the best single model on GSM8K, MMLU, and ARC.At inference, the image channel is masked, creating a zero-shot modality-transfer setting.

7. Conclusion

MMR-Bench is an offline, cost-aware benchmark for reproducibly evaluating routing across heterogeneous MLLMs. The experiments show that multimodal routing improves cost–accuracy trade-offs, with adaptive fusion and matrix-factorization routers supporting robust transfer.

  • Conclusion: MMR-Bench provides precomputed utilities, normalized costs, unified metrics, frozen splits, and deterministic scoring for controlled offline routing evaluation.The benchmark covers over 100k instance–model pairs across multiple candidates and tasks.
  • Conclusion: Routing sometimes matches or exceeds the best single model's accuracy at roughly one-third of the cost.Matrix-factorization-based routers achieve the most robust performance across heterogeneous workloads.
  • Conclusion: Adaptive multimodal fusion closes the gap left by unimodal policies, while learned routers remain stable under distribution shifts and transfer to text-only benchmarks.The benchmark is positioned as a standardized testbed for cost-aware multimodal routing and MLLM deployment.

Supplementary Material for MMR-Bench: A Comprehensive Benchmark for Multimodal LLM Routing

The supplementary material documents dataset metadata, model-zoo specifications, router and fusion implementations, and complete experimental results supporting MMR-Bench.

  • Supplementary material: The supplement organizes detailed dataset statistics and licensing, model-zoo and cost-normalization specifications, router and adaptive-fusion implementation details, and complete experimental results.It specifically points to Sections B–E for these materials.

B. Dataset Details and Composition

The supplementary dataset details describe standardized processing across eight datasets, a ten-model zoo spanning deployment settings, unified cost accounting, and frozen embedding-based routing features.

  • Dataset processing: MMR-Bench uses VLMEvalKit with dataset-native prompts, official chat templates, and model-specific image preprocessing.MCQ datasets include questions, options, and selection instructions, while VQA datasets use their original questions.
  • Cost accounting: Per-instance costs aggregate input-token, image-token, and output-token charges under a unified pricing scheme.Efficiency-oriented models disable long-form reasoning traces to reduce latency and cost.
  • Model zoo: The candidate zoo contains K = 10 open-weight and commercial MLLMs spanning capacities, architectures, training regimes, and deployment costs.The listed models include InternVL3, Qwen2.5-VL, Gemma3, GPT-5, Claude, and Gemini variants.
  • Cost normalization: Normalized cost balances commercial API pricing and open-weight inference latency by scaling each model relative to the most expensive raw cost.The raw-cost set is denoted by C.
  • Feature fusion: Routers operate on frozen text and image embeddings, with multimodal features formed through adaptive weighting and interaction terms.The fusion mechanism combines confidence-based modality weights with element-wise product and difference features before normalization.

D.3. Router Architectures and Hyperparameters

The router suite predicts per-model utility and cost from fused multimodal features, then selects among candidate models using cost-aware routing rules. It includes non-parametric, linear, neural, low-rank, matrix-factorization, attention-based, random, and oracle approaches.

  • Prediction targets and selection rule: Routers predict per-model utility and cost, then select a candidate using a trade-off weight λ.Utility is higher-is-better, while cost is lower-is-better.
  • Non-parametric routers: KMeansRouter assigns instances to fused-embedding clusters and uses cluster-level average utility and cost estimates.Test instances inherit predictions from their nearest cluster centroid.
  • Non-parametric routers: KNNRouter estimates each candidate’s utility and cost by averaging the k nearest neighbors in fused embedding space.Nearest-neighbor retrieval uses cosine distance.
  • Regression routers: LinearRouter and MLPRouter learn separate predictors for per-model utility and cost from multimodal embeddings.LinearRouter uses regressors, while MLPRouter uses independent two-layer MLPs trained with MSE and Adam.
  • Low-rank and matrix-factorization routers: Low-rank routers reduce fused features before regression or learn shared latent factors for instances and candidate models.LinearMFRouter uses TruncatedSVD and ridge regression; MLPMFRouter uses learned instance and model vectors with separate cost modeling.
  • Attention and reference routers: CrossModalRouter models modality interactions with stacked multi-head attention and predicts correctness and cost.RandomRouter supplies a lower bound, while OracleRouter uses ground-truth utilities and costs as an upper bound.

E.1. Quality-Neutral Cost (QNC) Analysis

QNC evaluates the cost required to reach the best single-model accuracy, while Pareto frontiers compare routing efficiency with static model choices. Learnable routing policies are generally at least as cost-efficient as the best single model, with performance varying by task family.

  • QNC definition: QNC < 1 means a router reaches target accuracy below the best single-model cost, while QNC = ∞ means it cannot reach that target.The metric directly measures quality-neutral cost relative to the best single model.
  • Overall QNC results: 0.970 average QNC on All is achieved by LinearMF, compared with 0.995 for MLPMF.Both learnable MF-based routers also achieve QNC < 1 on several benchmarks, including SEED-Bench-2-Plus, MathVerse, and MMStar.
  • Pareto frontiers: Routing policies form a convex hull above single-model baselines across six datasets, producing a better cost–accuracy Pareto frontier than static choices.The figure compares normalized cost on the x-axis with accuracy on the y-axis.
  • Task-dependent routing behavior: Instance-based methods show sharper low-cost gains on mathematical reasoning, whereas parametric routers scale more smoothly on general VQA and OCR.The comparison suggests task-family differences in which routing strategy best exploits structure or robustness.

F. Qualitative Case Studies

The case studies show routing cheaply handles clear OCR while escalating visually grounded, multi-step reasoning to a frontier model. These examples illustrate how routing distinguishes routine perception from capability-intensive multimodal reasoning.

  • Case A: Efficiency via Routing: Clear OCR with large unambiguous text is routed to Qwen2.5-VL-3B, avoiding unnecessary use of heavier models.The router identifies low difficulty and the smallest model answers correctly.
  • Case B: Multimodal Reasoning: The Kangaroo and Rabbit problem is routed to GPT-5 because it requires precise visual grounding and multi-step logical reasoning.The query is deceptively simple textually but depends on diagrammatic information and a reasoning chain.
  • Implication: The cases frame MMR-Bench’s value as distinguishing commodity tasks from premium tasks to optimize computational investment.The contrast links low-cost routine perception with frontier-capability reasoning demands.
  • Case B: Multimodal Reasoning: GPT-5 produces the correct answer “76” for the Kangaroo and Rabbit problem, while smaller models fail to ground the initial position or calculation.The example demonstrates capability-oriented escalation for difficult multimodal reasoning.
  • Figure S2 overview: Figure S2 contrasts a low-difficulty OCR decision with a high-capability mathematical reasoning decision.The visualization emphasizes how routing responds differently to task complexity.
Loading 2601.17814v1…