PaperlayerView exact revision on arXiv

Source-linked AI summary

Unchosen Experts Can Contribute Too: Unleashing MoE Models' Power by Self-Contrast

Chufan Shi, Cheng Yang, Xinyu Zhu, Jiahao Wang, Taiqiang Wu, Siheng Li, Deng Cai, Yujiu Yang, Yu Meng

arXiv:2405.14507v2cs.CLcs.LG

TL;DR

MoE leaves most experts unused, while simply activating more experts does not reliably improve output quality. SCMoE contrasts strong and weak routing outputs from the same model, and improves Mixtral 8x7B performance across multiple benchmarks with only a minor inference-latency increase.

  • Problem

    MoE inference activates only a portion of the experts, leaving the potential contribution of unchosen experts underexplored.

    Source: §1 · p. 2

  • Method

    SCMoE determines next-token probabilities by contrasting outputs from strong and weak activation routing strategies within the same MoE model.

    Source: §1 · p. 2, §2.3 · p. 5

  • Results

    SCMoE improves Mixtral 8x7B across multiple benchmarks; GSM8K accuracy rises from 61.79 to 66.94 (+5.15) versus greedy decoding.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Self-contrast can use information from different routing strategies to enhance MoE reasoning performance with a minor inference-time latency increase.

    Source: §6 · p. 9

    Resource constraints prevented evaluation on larger MoE models such as Mixtral 8x22B or DeepSeek-V2.

    Source: §6 · p. 9

Abstract

from arXiv · show

Mixture-of-Experts (MoE) has emerged as a prominent architecture for scaling model size while maintaining computational efficiency. In MoE, each token in the input sequence activates a different subset of experts determined by a routing mechanism. However, the unchosen experts in MoE models do not contribute to the output, potentially leading to underutilization of the model's capacity. In this work, we first conduct exploratory studies to demonstrate that increasing the number of activated experts does not necessarily improve and can even degrade the output quality. Then, we show that output distributions from an MoE model using different routing strategies substantially differ, indicating that different experts do not always act synergistically. Motivated by these findings, we propose Self-Contrast Mixture-of-Experts (SCMoE), a training-free strategy that utilizes unchosen experts in a self-contrast manner during inference. In SCMoE, the next-token probabilities are determined by contrasting the outputs from strong and weak activation using the same MoE model. Our method is conceptually simple and computationally lightweight, as it incurs minimal latency compared to greedy decoding. Experiments on several benchmarks (GSM8K, StrategyQA, MBPP and HumanEval) demonstrate that SCMoE can consistently enhance Mixtral 8x7B's reasoning capability across various domains. For example, it improves the accuracy on GSM8K from 61.79 to 66.94. Moreover, combining SCMoE with self-consistency yields additional gains, increasing major@20 accuracy from 75.59 to 78.31.

1 Introduction

MoE leaves most experts unused, and simply activating more experts can hurt performance. SCMoE contrasts outputs from strong and weak routing strategies, improving Mixtral 8x7B results across several benchmarks.

1 Introduction

  • MoE reduces computation by activating only selected experts per input token, leaving most trained experts out of each prediction.The potential of using those unchosen experts at inference to improve performance remains underexplored.Source: §1 · pp. 1–2
  • Simply increasing activated experts on Mixtral 8x7B does not yield stable gains and can hurt performance across tasks.The exploratory results challenge the assumption that incorporating more experts necessarily improves performance.Source: §1 · p. 2
  • Different routing strategies produce divergent token distributions, especially in rigorous reasoning, suggesting experts may conflict rather than act synergistically.This motivates using their differences rather than assuming additional experts will cooperate.Source: §1 · p. 2
  • SCMoE determines next-token probabilities by contrasting logits from strong top-2 activation with weak rank-k activation, allowing unchosen experts to contribute.The method is presented as converting negative effects from unchosen experts into positive ones through the contrast.Source: §1 · p. 2
  • SCMoE improves Mixtral 8x7B across GSM8K, StrategyQA, MBPP, and HumanEval; GSM8K accuracy rises from 61.79 to 66.94 (+5.15).Combining SCMoE with self-consistency raises GSM8K major@20 accuracy from 75.59 to 78.31 (+2.72); inference delay is x1.30 versus greedy decoding.Source: §1 · p. 2

2 Method

The method examines how different routing strategies use MoE experts and proposes SCMoE, which contrasts strong and weak activation to use information from unchosen experts during inference.

2.1 Preliminary

  • Each MoE layer routes tokens to selected experts, renormalizes their gate weights, and combines their outputs as a weighted sum.The router first computes gate values from each token’s hidden state.Source: §2.1 · p. 3

2.2 Divergence Between Different Routing Strategies: An Exploratory Analysis

  • On GSM8K, top-2 and rank-k distributions diverge increasingly for k=2–8, especially on reasoning tokens, while function-word and punctuation predictions show smaller divergence.Rank-1 routing is relatively similar to top-2 routing; the analysis compares next-token distributions using KLD.Source: §2.2 · p. 4
  • Different routing strategies can exhibit conflicting generation behavior on reasoning predictions, motivating contrastive use of their information.The paper proposes leveraging differences such as those between top-2 and rank-k routing during inference.Source: §2.2 · pp. 4–5

2 Method

  • The method introduces SCMoE, which determines next-token distributions by contrasting strong top-2 activation with weak rank-k activation.The approach uses the contrast to amplify desirable behavior from strong activation.Source: §2.3 · p. 5

2.3 SCMoE: Self-Contrast Mixture-of-Experts

  • SCMoE adjusts next-token logits using strong- and weak-activation outputs, with β controlling the contrastive penalty and α controlling the valid-vocabulary set.The method uses top-2 routing for strong activation and rank-k routing for weak activation; α is empirically set to 0.1.Source: §2.3 · p. 5

3 Experiments

The experiments evaluate SCMoE on reasoning and code-generation benchmarks with Mixtral 8x7B, comparing it with routing-based and search-based baselines. SCMoE improves results across all four tasks, while the other search-based methods do not show substantial gains.

3.1 Datasets and Models

  • The evaluation uses Mixtral 8x7B on GSM8K and StrategyQA for reasoning and MBPP and HumanEval for code generation.The paper reports accuracy for reasoning tasks and pass@1 accuracy for code generation.Source: §3.1 · p. 5

3.2 Setup

  • Baselines include dynamic and ensemble routing, plus contrastive search, contrastive decoding, and DoLa.Dynamic routing varies the number of activated experts using a threshold; ensemble routing activates k experts.Source: §3.2 · pp. 5–6

3.3 Results

  • +5.10 GSM8K, +3.46 StrategyQA, +2.60 MBPP, and +7.92 HumanEval: SCMoE improves over greedy across all four tasks.Dynamic routing and ensemble routing also improve some tasks, but search-based baselines show no substantial improvements overall.Source: Table 1 · p. 6, §3.3 · p. 6

4 Analysis

The analysis finds that SCMoE’s results depend on activation and decoding choices, and reports gains from adjusting strong activation, combining SCMoE with self-consistency, and applying it to DeepSeekMoE-16B. It also measures decoding latency against other methods.

4.1 Impact of Weak Activation

  • SCMoE improves on greedy across GSM8K, StrategyQA, HumanEval, and MBPP with each tested weak activation, while rank-2 gives consistently good performance.Even random-1 routing as weak activation improves over greedy; the strong activation is top-2 in these experiments.Source: §4.1 · pp. 6–7, Figure 4 · p. 7

4.2 Impact of Strong Activation

  • Compared with top-2 strong activation, task-specific stronger routing improves performance by 1.98 on GSM8K, 0.13 on StrategyQA, and 1.80 on MBPP.The selected strong activation is top-3 for GSM8K and top-4 for StrategyQA, MBPP, and HumanEval; weak activation remains rank-2.Source: Table 2 · p. 7, §4.2 · p. 7

4.3 Combining SCMoE with Self-Consistency

  • SCMoE with self-consistency raises GSM8K major@20 accuracy from 75.59 to 78.31 (+2.72).SCMoE alone scores 67.94, comparable to major@5 at 66.87.Source: Figure 5 · p. 7, §4.3 · p. 8

4.4 Latency

  • SCMoE increases decoding time by 1.30x versus greedy, compared with 1.62x for contrastive search and 1.43x for contrastive decoding.The comparison measures forced generation of 512 tokens after a 32-token input.Source: Table 3 · p. 8, §4.4 · p. 8

4.5 Employ DeepSeekMoE

  • On DeepSeekMoE-16B, SCMoE improves over greedy by 2.00 on MBPP and 1.22 on HumanEval; other routing-based and search-based methods struggle to outperform the greedy baseline.Source: Table 4 · p. 8, §4.5 · p. 9

5 Related Work

The related work situates SCMoE among MoE efficiency and routing research and among contrastive methods for language modeling. It distinguishes SCMoE as an inference-time approach that contrasts activation strategies within one MoE model.

Mixture-of-Experts

  • MoE research uses sparse expert activation to balance model capacity and efficiency, with architectures including Mixtral, Grok-1, DeepSeekMoE, and QwenMoE.The cited models differ in routing and expert segmentation choices.Source: §5 · p. 9
  • Prior work examines MoE pruning and dynamic routing to reduce computation while maintaining performance; SCMoE instead uses unchosen experts through self-contrast.The paper frames this as boosting MoE capability without increasing computation too much.Source: §5 · p. 9

Contrast in Language Modeling

  • Unlike contrastive methods that compare different models or layers, SCMoE contrasts strong and weak activation within an MoE model at inference time.The related work also surveys training-time and inference-time contrastive language-modeling approaches.Source: §5 · p. 9

6 Conclusion

SCMoE uses self-contrast to leverage divergent information from different routing strategies, improving MoE reasoning with minor inference latency. The study could not test larger MoE models because of resource constraints.

6 Conclusion

  • SCMoE uses divergent outputs from different routing strategies in self-contrast to enhance MoE models’ reasoning in next-token prediction.The conclusion describes the strategy as conceptually simple and computationally lightweight.Source: §6 · p. 9
  • SCMoE improves performance on multiple benchmarks with only minor latency increase at inference time.Source: §6 · p. 9
  • Resource constraints prevented evaluation on larger MoE models such as Mixtral 8x22B or DeepSeek-V2.Source: §6 · p. 9

A.1 KLD Supplement for Section 2.2

The supplement measures next-token distribution divergence between top-2 and rank-k routing on GSM8K answers, finding larger differences for reasoning tokens than stopwords.

A.1 KLD Supplement for Section 2.2

  • The study compares top-2 routing with rank-k routing across all answer tokens, mathematical-expression tokens, and stopword tokens.It uses questions and ground-truth answers from the GSM8K training set and reports results in Table 5.Source: §A.1 · p. 13
  • As k increases from 2 to 8, average KLD between rank-k and top-2 routing increases, indicating greater overall next-token prediction discrepancy.The distributions are evaluated using Mixtral 8x7B on ground-truth GSM8K answers under teacher forcing.Source: §A.1 · p. 13
  • For every rank-k strategy, average KLD is apparent for mathematical-expression tokens, indicating routing differences when generating reasoning tokens.Source: §A.1 · p. 13
  • For every rank-k strategy, stopword-token KLD is relatively smaller than mathematical-expression-token KLD, suggesting fewer challenges for rank-k routing on stopwords.Source: §A.1 · p. 13

A.2 Further Analysis on Kullback-Leibler Divergence

Rank-2 routing differs more from top-2 routing on expression tokens than on all tokens, while Mistral 7B’s similar output distributions make it unsuitable as a weak model for contrast.

A.2 Further Analysis on Kullback-Leibler Divergence

  • 31.13%: expression-token KLD exceeds all-token KLD when shifting from top-2 to rank-2 routing, indicating a larger reasoning-capability decrease than basic-generation decrease.Source: §A.2 · p. 14
  • Rank-2 routing is the main weak-activation choice because contrast with rank-2 generally yields better improvements than rank-3, rank-4, or random-1 routing.The same reasoning-ability gap is observed with those alternatives, with varying significance.Source: §A.2 · p. 14
  • Mistral 7B is not an ideal weak model to contrast because its next-token distributions are quite similar to Mixtral 8x7B with top-2 routing.Source: §A.2 · p. 14

B Quantitative Study of SCMoE’s Unchosen Experts’ Utilization.

On GSM8K with Mixtral 8x7B, rank-k weak activation selects many experts that top-2 strong activation does not select.

B Quantitative Study of SCMoE’s Unchosen Experts’ Utilization

C Hyperparamters Setting for DeepSeekMoE-16B

The section specifies DeepSeekMoE-16B baseline settings and SCMoE’s activation and penalty search ranges. DeepSeekMoE-16B lacks a smaller same-vocabulary model for a contrastive-decoding baseline.

D Detailed Results of Different Hyperparamters Setting for Each Method

The appendix details decoding-method hyperparameter results and explains the scope of SCMoE’s effectiveness. It reports a DoLa generation failure on Mixtral and cautions that SCMoE may offer no distinct advantage on benchmarks without explicit reasoning paths.

D Detailed Results of Different Hyperparamters Setting for Each Method

E Failed Examples of DoLa on Mixtral 7x8B

F Scope of SCMoE’s Effectiveness

Related papers

Loading 2405.14507v2…