Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts

1 Introduction

• MoE models use sparse expert activation to handle visual-textual interactions efficiently, but this success leaves open how they integrate perception and reasoning.The introduction frames this as the central puzzle motivating the study.Source: §1 · p. 1
• Seeing but Not Thinking occurs when a model extracts image content accurately but reasons incorrectly, while solving the identical problem presented as text.The paper poses the question of what causes this visual-input reasoning failure.Source: §1 · p. 1
• In controlled MATH500 experiments, 68.2% to 73.1% of text-success/image-failure cases stem from reasoning errors, and cross-modal concept interventions confirm semantic sharing in MoE models.Intervention success follows an inverted U across layers, peaking in the middle; the finding rules out semantic alignment failure as the sole explanation.Source: §1 · p. 2
• Visual experts concentrate in early and terminal layers, while domain-specific reasoning experts cluster in middle layers where image inputs induce routing divergence.Greater routing divergence correlates with lower reasoning accuracy; the Routing Distraction hypothesis attributes failures to insufficient activation of task-relevant experts, not impaired intrinsic reasoning.Source: §1 · p. 2
• Routing-guided intervention consistently improves three MoE models across six benchmarks, with gains up to 3.17% on complex visual reasoning tasks.The authors also report that identified domain experts transfer across tasks with different information structures when text references elicit the target reasoning patterns.Source: §1 · p. 2

Multimodal Semantic Sharing

• Prior work finds that visual and text tokens activate similar LLM weights despite distinct representations, while semantically equivalent text-image inputs can align into modality-invariant task vectors (Shukor and Cord, 2024; Luo et al., 2024).Source: §2 · p. 3

MoE Expert Specialization

• Research on MoE specialization associates experts with cognitive operations, safety behavior, and domains, with differentiation increasing across layer depth.The cited work includes Wang et al. (2025a), Lai et al. (2025), Xue et al. (2024), Li et al. (2025a), and Lo et al. (2025).Source: §2 · p. 3

Routing Intervention

• Inference-time routing methods refine expert selection, contrast selected and unselected expert outputs, adjust routing logits, or activate more experts for complex tasks.Examples include R2-T2, SC-MoE, SteerMoE, and dynamic routing approaches.Source: §2 · p. 3

3.1 Cross-Modal Semantic Sharing in MoE

• Middle-layer intervention success exceeds 90%, showing cross-modal semantic sharing and ruling out semantic alignment failure as the sole explanation.Success follows an inverted U-shaped pattern across layers, with low rates early and a sharp decline in terminal layers.Source: §3.1 · p. 4

3.2 Layer-wise Expert Specialization

• Math experts cluster in middle layers, while visual experts concentrate in early and terminal layers with minimal overlap in the middle.Math experts are identified using GSM8K versus Alpaca with τ = 0.3; visual experts use image-versus-text Alpaca inputs with τ = 0.2.Source: §3.2 · p. 4, Figure 4 · p. 5

3.3 Routing Divergence Across Modalities

• Visual complexity primarily affects routing in middle layers (6–42), where domain experts concentrate, while early and terminal layers show larger image-text divergence overall.The divergence curves for the three image versions differ mainly in the middle-layer region.Source: §3.2 · p. 4, §3.3 · p. 5, Figure 3 · p. 5
• 89.0%/88.2%/87.4% reasoning accuracy for image versions v1/v2/v3, versus 92.8% for pure text, correlates with greater middle-layer JSD as accuracy falls.The three versions have similar perception error rates; the authors suggest middle-layer routing divergence, rather than perceptual quality, contributes to reasoning degradation.Source: §3.3 · p. 5

4.1 Domain Expert Identification

• Domain experts are identified by comparing activation rates on domain-specific data and Alpaca, using text references suited to each visual scenario.Semantically equivalent scenarios use the original text problems; natural visual scenarios use task-appropriate proxies such as text-only versions or model-generated descriptions.Source: §4.1 · p. 6
• Only 20 randomly sampled examples are needed to identify domain experts, and the samples need not be solved or strictly equivalent to the visual task.The procedure computes each expert's activation-frequency difference and applies threshold τ.Source: §4.1 · p. 6

4.2 Routing Weight Adjustment

• At inference, soft intervention moderately boosts domain-expert logits while preserving router flexibility; hard intervention sets them to the layer maximum.The random baseline enhances an equal number of randomly selected experts using the soft strategy, controlling for routing perturbation itself.Source: §4.2 · p. 6

5.2 Main Results

• In semantically equivalent image-rendered tasks, Kimi-VL rises from 52.30% to 54.54% (+2.24%) on math and from 29.51% to 32.49% (+2.98%) on physics.Source: §5.2 · p. 7
• Soft intervention improves reasoning across models and scenarios, reaching +3.17% for Kimi-VL on MathVerse.Source: §5.2 · pp. 6–7, Table 2 · p. 8
• On natural visual tasks, Qwen3-VL improves from 55.92% to 57.57% (+1.65%) on MATH-Vision, while GSM8K-V gains are modest when perception dominates task difficulty.Source: §5.2 · p. 7
• Soft intervention is the most consistent strategy; hard intervention can severely degrade Llama4-Scout, while random routing perturbation generally yields no improvement.The authors attribute Llama4’s degradation to its Top-1 routing mechanism, where forcing expert logits can disrupt original routing decisions.Source: §5.2 · p. 7

5.3 Analysis

• Partial MathVerse text references still support transfer: Qwen3-VL vision-only accuracy rises from 69.29% to 71.20% (+1.91%) with matched guidance.The authors conclude that expert identification locates domain-specific cognitive functions rather than sample-specific solution paths.Source: §5.3 · p. 7
• Guidance is domain-sensitive: Qwen3-VL gains +1.91% with MathVerse-matched experts but only +0.25% with GSM8K-mismatched experts; Llama4 changes from +1.02% to -1.01%.The cited comparison distinguishes arithmetic-focused GSM8K from MathVerse’s geometric reasoning and function analysis.Source: §5.3 · p. 7, Table 3 · p. 8
• Intervention layers are architecture-dependent: Qwen3-VL and Llama4-Scout favor middle layers, while Kimi-VL benefits from early-plus-middle layers.Source: §5.3 · pp. 7–8, Table 4 · p. 8
• Kimi-VL and Qwen3-VL perform best with λ ∈[0.4, 0.6], whereas Llama4 requires λ = 0.2; excessive strength can override input-specific routing.Figure 5 examines the effect of enhancement coefficient λ; the authors link Llama4’s weaker setting to its Top-1 routing mechanism.Source: §5.3 · p. 8, Figure 5 · p. 8

• The analysis finds visual and domain experts separated by layer, with image-induced middle-layer routing divergence correlated with reasoning degradation.Source: §6 · p. 8
• Routing-guided intervention enhances domain-expert activation and yields consistent improvements across three models and six benchmarks, with gains up to 3.17% on complex visual reasoning tasks.Source: §6 · p. 9
• The paper presents mechanistic insights into multimodal reasoning failures alongside a practical mitigation method.Source: §6 · p. 9

Limitations

• Routing guidance cannot remedy perception errors and targets only cases where perception is correct but reasoning fails.It does not address the broader challenge of visual understanding.Source: Limitations · p. 9
• The approach requires task-specific text references and empirical selection of intervention layers and strengths for each model-task combination.The authors identify automatic expert identification and parameter calibration as future work for improving practical applicability.Source: Limitations · p. 9
• Cross-modal alignment is verified for simple numerical concepts, while alignment quality for complex visual concepts remains unclear and may be another source of reasoning failures.Source: Limitations · p. 9
• Routing divergence and reasoning degradation are correlated, but establishing strict causality requires more controlled interventions.Improvements from routing guidance provide supporting evidence, while the causal mechanisms remain under investigation.Source: Limitations · p. 9

A Discussion

• Routing Distraction addresses cases where perception succeeds but reasoning fails, not perceptual errors or all multimodal reasoning failures.Guidance provides limited benefit when difficulty centers on perception, as in GSM8K-V tasks requiring extraction across multiple scene images.Source: A Discussion · p. 12
• Three models with Top-K and Top-1 routing show consistent improvements, but optimal intervention layers and strengths vary across architectures.Source: A Discussion · p. 12
• Routing guidance is a post-hoc remedy requiring task-specific expert identification and tuning; the discussion proposes training-time routing objectives as a more fundamental direction.Suggestions include regularizing modality-dependent expert selection or aligning routing distributions across semantically equivalent inputs.Source: A Discussion · p. 12

B Case Study

• In the GSM8K case, the model extracted the needed information but miscounted the initial value as a separate day; routing guidance corrected the count from Day 0.Source: B Case Study · p. 12

C Construction and Validation of Semantically Equivalent Dataset

• MATH500 problems were rendered as three image styles: printed text, handwritten text, and handwritten text on a light-gray grid.The images were high-resolution LaTeX renderings; versions of each problem were padded to identical dimensions.Source: C Construction and Validation of… · p. 12

Prompt for OCR First Strategy

• Adding an OCR-first instruction did not improve results, so the authors instead analyzed directly supplied images.Source: Prompt for OCR First Strategy · p. 13

Prompt for Error Classification

• Error analysis compared image answers with correct text solutions, distinguishing information-reading errors from reasoning or computation errors.Table 6 compares accuracy and error percentages on MATH500 and its three image versions; the error percentages use text-correct, image-incorrect samples.Source: Prompt for Error Classification · pp. 13–14, Table 6 · p. 14

D Model Information

• The appendix points to a table of architectural specifications for the three evaluated MoE models, with expert counts specified per layer.Source: Table 7 · p. 14

E.1 Benchmark Details

• MATH500 and GPQA-Diamond problems were rendered as high-resolution images using the v2 approach to create semantically equivalent visual inputs.The evaluated GPQA-Diamond subsets were physics and chemistry; the biology subset was excluded because it contained only 19 questions.Source: §E.1 · p. 14
• MathVerse evaluation uses its vision-only version, comprising 788 test samples across geometry and function-graph problem types.Source: §E.1 · p. 14
• MATH-Vision contains 3,040 problems with visual contexts from math competitions, and evaluation uses its 304-sample mini test set.Source: §E.1 · p. 14
• GSM8K-V converts text problems into a visual multi-image format, and evaluation uses the implicit version to maximize visual dependency.The dataset comprises 1,319 samples and 5,343 images requiring information extraction from scene images.Source: §E.1 · p. 14

E.2 MathVision Description Generation

• For MATH-Vision, detailed diagram descriptions are generated as text references for domain-expert identification and concatenated with the problem text.Problem types whose answers are directly evident in the description, such as counting, are excluded; the prompt requests a complete diagram description.Source: §E.2 · p. 14, Prompt for MathVision Description… · pp. 14–15

• Experiments used temperature=0 greedy decoding, zero-shot prompting, and a maximum generation length of 8192 tokens, with no external tools.Source: F Implementation Details · p. 15
• EasySteer wrapped the vLLM MoE router to intercept and modify routing logits during inference; experiments ran on 16 A100 GPUs using vLLM v0.11.0.Source: F Implementation Details · p. 15
• For Qwen3-VL-30B-A3B-Instruct, repetition_penalty=1.2 was used on all benchmarks except MATH500 to mitigate excessively long chains of thought, attributed likely to long-CoT cold-start training.Source: F Implementation Details · p. 15