PaperlayerView exact revision on arXiv

Source-linked AI summary

MHR-Router: Multimodal Heterogeneous Retrieval-Enhanced Adaptive Routing for VLM Inference

Wenjuan Zhu, Zheming Yang, Yunqing Hu, Qi Guo

arXiv:2610.13278v1cs.LG

TL;DR

VLM routing must choose among experts whose strengths vary across image-question queries, while text-based cues and global profiles can miss visual evidence and local failure patterns. MHR-Router fuses frozen multimodal features and retrieves expert-specific success and failure evidence for a shared scorer. Across its evaluation, it reports higher routing accuracy than a development-selected best single expert and a matched retrieval-free multimodal router.

  • Problem

    Expert strengths vary by image-question query, while text-based routing can miss visual evidence and global profiles can obscure query-level success and failure patterns.

    Source: §1 · p. 1

  • Method

    MHR-Router fuses frozen image-text features into a routing representation and uses retrieved expert-specific success and failure memories as local evidence for a shared scorer.

    Source: §1 · p. 1

  • Results

    71.58±0.02% routing accuracy on 6,400 queries exceeds the development-selected best single expert by 7.36% and a matched retrieval-free multimodal router by 1.22%.

    Source: §1 · p. 1

  • Takeaways & Limitations

    Across eight abilities and eight experts, MHR-Router supports input-dependent expert selection without ability or source-task labels at inference.

    Source: §5 · p. 4

Abstract

from arXiv · show

Vision-language models (VLMs) exhibit complementary capabilities, yet selecting the most suitable expert for a given image-question query remains challenging due to heterogeneous multimodal inputs and varying expert strengths. Existing routing approaches rely on text-based cues or static model profiles, failing to capture fine-grained visual evidence and expert-specific failure patterns. To address this, we propose MHR-Router, a multimodal heterogeneous retrieval-enhanced adaptive routing framework for VLM inference. It accurately dispatches each multimodal query to the most suitable VLM expert via local retrieval evidence. MHR-Router consists of two stages: (1) Multimodal Routing Representation, which fuses frozen multimodal features into a comprehensive query representation; and (2) Heterogeneous Retrieval-Enhanced Adaptive Routing, which retrieves expert-specific success and failure memories as local evidence and employs a shared scorer to adaptively select the optimal expert. Experiments show that MHR-Router effectively achieves heterogeneous modality routing and consistently improves accuracy over traditional baselines.

1. INTRODUCTION

VLM routing must select an expert from joint image-question evidence, since expert strengths vary by query and text-only cues can miss visual information. MHR-Router combines frozen multimodal features with expert-specific local outcome memories, and its experiments report higher routing accuracy than both a development-selected single expert and a matched retrieval-free router.

1. INTRODUCTION

  • Expert strengths vary across image-question pairs, making fixed selection leave complementary capabilities unused while evaluating every candidate requires invoking the full expert pool.Routing estimates suitability before any candidate expert is invoked.Source: §1 · p. 1
  • Text-only routers use prompt semantics, but VLM routing must also consider visual evidence that may be absent from the question.Figure 1 contrasts prompt-semantic LLM routing with image-and-question VLM routing.Source: §1 · p. 1, Figure 1 · p. 1
  • MHR-Router fuses frozen image-text features into a unified routing representation and retrieves expert-specific success and failure evidence for a shared scorer.The design targets global profiles that can obscure where an expert succeeds or fails across heterogeneous queries.Source: §1 · p. 1
  • 71.58±0.02% routing accuracy exceeds the development-selected best single expert by 7.36% and a matched retrieval-free multimodal router by 1.22%.The evaluation spans 6,400 queries, eight visual abilities, and eight open-source VLMs.Source: §1 · p. 1

2. RELATED WORK

Prior routing methods primarily target text-only LLM selection, while VLM routing must account for visual evidence and query-level differences in expert outcomes.

2. RELATED WORK

  • Text-based routing methods and multimodal-query VLM selection have advanced, but global profiles can miss expert-specific failure patterns and fine-grained query-level evidence.Prior work includes ARMS, which uses multimodal queries and VLM profiles [9].Source: §2 · p. 2

3. METHOD

MHR-Router first represents each image-question query with frozen multimodal features, then uses expert-specific success and failure memories as local evidence for a shared scorer to rank candidate experts.

3.1. Multimodal Routing Representation

  • Frozen multimodal features represent image-question queries for expert ranking without invoking a candidate VLM.The feature extractor remains fixed during router training, preserving a reusable representation shared with the retrieval memories.Source: §3.1 · p. 2

3.2. Heterogeneous Retrieval-Enhanced Adaptive Routing

  • Each expert's training-query representations are partitioned into nonparametric success and failure memories, forming a profile of its competence in the frozen representation space.The two memories provide expert-specific evidence without adding learnable expert parameters.Source: §3.2 · p. 2
  • For each expert, retrieval summarizes success and failure neighborhoods with mean and maximum similarities and contrasts the two kinds of evidence.Training queries exclude themselves from retrieval, so their own correctness labels do not provide their evidence.Source: §3.2 · pp. 2–3
  • A shared scorer combines the query representation, expert identity, and retrieved evidence to rank experts; only the selected expert receives the original query.At inference, the router uses no test correctness, ability, source-task, or global expert-accuracy labels or features.Source: §3.2 · p. 3

4. EXPERIMENTS

Across a 6,400-query, eight-ability benchmark, MHR-Router combines multimodal representations with retrieval evidence and outperforms fixed and retrieval-free references. Further results examine modality choices, ability-level performance, parameter sensitivity, and retrieval evidence groups.

4.1. Experimental Setup

  • The benchmark spans 6,400 queries across eight visual abilities and eight VLMs; image-disjoint splits yield 3,820 test queries, with accuracy based on selected experts’ judged responses.Each test query is routed to exactly one expert.Source: §4.1 · p. 3

4.2.1. Effect of Multimodal Query Representations

  • Image-plus-question SigLIP routing reaches 71.19%, versus 67.95% with text-only features; visual evidence contributes more than encoder choice, while questions add ability-specific gains.Qwen2.5-VL has the highest overall encoder mean, but no evaluated encoder dominates all abilities.Source: Figure 3 · p. 3, §4.2.1 · pp. 3–4

4.2.2. Routing Accuracy

  • MHR-Router reaches 71.58±0.02%, exceeding Best Single by 7.36% and the matched Multimodal Query Router by 1.22%.The Multimodal Query Router uses the same query embedding, expert identity, scorer, objective, and training schedule, but sets retrieval statistics to zero.Source: Table 1 · p. 4, §4.2.2 · p. 4
  • MHR-Router improves five of eight abilities, matches counting, and gains most in fine-grained recognition (+21.60 pp) and defect anomaly detection (+8.54 pp).The comparisons are against the development-selected expert for each ability.Source: Table 2 · p. 4, §4.2.2 · p. 4

4.2.3. Parameter Sensitivity Analysis

  • The study supports k = 5, λ = 0.5, and γ = 0.2 as stable defaults; mean accuracy varies by at most 0.23% across tested γ values.Accuracy rises from 70.83±0.10% at k = 1 to 71.58±0.02% at k = 5, then changes by at most 0.14%; λ = 0.5 yields the highest mean and lowest variation.Source: Figure 4 · p. 4, §4.2.3 · p. 4

4.3. Ablation Study

  • MHR-Router reaches 71.58±0.02%, versus 70.41±0.54% without positive evidence, 70.34±0.28% without negative evidence, and 70.36±0.38% without retrieval evidence.The full model exceeds every ablation in all three seeds, supporting joint use of success, failure, and contrast evidence.Source: §4.3 · p. 4

5. CONCLUSION

MHR-Router combines frozen multimodal representations with expert-specific success and failure memories, outperforming two routing baselines across eight abilities and eight experts.

5. CONCLUSION

  • Across eight abilities and eight experts, MHR-Router outperforms the best single expert selected on the development set by 7.36% and the matched retrieval-free Multimodal Query Router by 1.22%.Source: §5 · p. 4
  • MHR-Router combines frozen multimodal representations with expert-specific success and failure memories for adaptive VLM selection.Source: §5 · p. 4
  • At inference time, MHR-Router enables input-dependent expert selection without requiring ability or source-task labels.Source: §5 · p. 4

Related papers

Loading 2610.13278v1…