PaperlayerView exact revision on arXiv

Source-linked AI summary

CertMoE: Robustness Certification for Mixture-of-Experts LLMs via Routing-Aware Smoothing

Qingyue Wang, Yudong Gao, Lixia Wang, Ruixuan Huang, Yuanyuan Yuan, Yangqiu Song, Shuai Wang

arXiv:2610.29395v1cs.CRcs.SE

TL;DR

Existing robustness certifiers assume fixed computation graphs, whereas MoE routing can change the active experts under input perturbations. CERTMOE certifies routing containment and sub-network stability; experiments report substantially fewer violations than existing methods.

  • Problem

    Existing certifiers assume a fixed computation graph, but MoE routing changes the activated experts under perturbations, making the certified function input-dependent.

    Source: §I · p. 1, §III · p. 3

  • Method

    CERTMOE certifies relaxed routing containment, then prediction stability of the induced sub-network, and combines both certificates with a union bound.

    Source: §I · p. 2

  • Results

    Violation rate (VR) exceeded 30% on SST-2 for existing methods, while CERTMOE substantially reduced violations.

    Source: §I · pp. 1–2

  • Takeaways & Limitations

    Experiments show CERTMOE produces more reliable certificates by explicitly accounting for routing uncertainty in MoE LLMs.

    Source: §VII · p. 11

Abstract

from arXiv · show

Provable robustness is a cornerstone of trustworthy machine learning, aiming to guarantee model predictions under adversarial perturbations. Existing certification frameworks, inspired by abstract interpretation or symbolic execution, typically assume a fixed computation graph, where input perturbations or variations affect only model outputs but not the underlying execution path. This assumption, however, is violated in Mixture-of-Experts (MoE) Large Language Models (LLMs), whose input-dependent routing induces discrete and dynamic execution paths. Crucially, this dynamic computation changes the certified object from a single fixed function to a composition of input-dependent sub-functions, leading to unreliable certification guarantees. This paper presents \textsc{CertMoE}, the first routing-aware robustness certification framework for MoE LLMs, which explicitly accounts for input-induced routing dynamics in certification. Specifically, we introduce a relaxed routing-containment criterion, which certifies that perturbed Top-$K$ routing is contained within a larger certified expert pool. Based on this guarantee, \textsc{CertMoE} then constructs the corresponding induced sub-network and certifies its prediction consistency. This hierarchical certification procedure yields a joint probabilistic guarantee over routing containment and sub-network robustness, providing sufficient conditions under which the original prediction remains invariant. Extensive experiments across recent MoE LLMs demonstrate that dynamic routing makes certificates issued by existing methods frequently violated, while \textsc{CertMoE} reduces the empirical violation rate with routing-aware guarantees, e.g., from more than $30\%$ to $0$, yielding certificates that more faithfully reflect the robustness of MoE models.

I. INTRODUCTION

Existing certifiers assume a fixed computation graph, but MoE routing can change the active experts under perturbation, making certificates unreliable. CERTMOE accounts for routing changes and substantially reduces observed certificate violations.

I. INTRODUCTION

  • MoE routing can change the activated expert set within a certified region, so fixed-graph methods may certify different input-dependent functions and issue arbitrary certificates.The paper reports that standard attacks invalidate more than 30% of existing-method certificates on SST-2.Source: §I · p. 1
  • CERTMOE certifies relaxed routing containment and induced sub-network prediction stability, then combines the two certificates with a union bound for an end-to-end guarantee.The relaxed expert pool covers perturbation-induced execution paths without requiring exact routing agreement.Source: §I · p. 2
  • Violation rate (VR) exceeded 30% on SST-2 for existing methods, while CERTMOE substantially reduced violations.The violation rate measures issued certificates for which standard attacks find adversarial examples within the certified region.Source: §I · pp. 1–2
  • Experiments use RanMASK and SelfDenoise across two classification tasks and three representative MoE LLMs; pool size trades certificate assignment against possible violations.The framework also supports failure attribution and adds roughly 2× runtime only when certification is requested, leaving standard inference unchanged.Source: §I · p. 2

II. BACKGROUND AND RELATED WORK

MoE layers route tokens to selected experts, while text robustness certification commonly uses randomized smoothing over masked inputs. Because perturbations can also change routing, the paper motivates routing-aware certification and evaluates it as an on-demand tool for text classification.

II. BACKGROUND AND RELATED WORK

  • A Transformer MoE layer replaces the FFN with a router and multiple experts; each token activates its Top-K experts, so inputs can follow different computation paths.The router computes a gating distribution over experts and selects the Top-K ranked experts.Source: §II · pp. 2–3
  • Text randomized smoothing masks a randomly selected portion of tokens, aggregates predictions over sampled masks, and can certify label invariance within a perturbation radius.The masking procedure keeps a fixed number of token positions and replaces the remainder with [MASK].Source: §II · p. 3

III. MOTIVATION AND APPLICATION SCOPE

  • In MoE models, perturbations can change Top-K routing and thus which expert-subset function is being certified; the cited layer-wise analysis reports flips for over 50% of tokens.The analysis covers DeepSeek-V2-Lite and Qwen1.5-MoE-A2.7B.Source: §III · p. 3
  • CERTMOE is evaluated on text-only MoE LLM classification with label-invariance certification and is invoked on demand, while standard inference remains unchanged.The paper notes that the framework may extend to other tasks and modalities.Source: §III · p. 3

IV. CERTMOE: ROUTING-AWARE MOE CERTIFICATION

CERTMOE separates routing certification from prediction certification, then combines them to certify MoE predictions. Its routing stage uses smoothed, sentence-level routing to certify a larger expert pool that can contain perturbed Top-K selections.

IV. CERTMOE: ROUTING-AWARE MOE CERTIFICATION

  • CERTMOE first certifies that perturbed Top-K routing stays within certified expert pools, then applies an output-level certifier to the induced sub-network.This modular design allows the prediction certifier to be instantiated with existing output-level methods.Source: §IV · p. 4

A. Routing Containment Certification

  • Token-level routing is aggregated into a sentence-level decision, avoiding separate certification of every token’s routing choice.The counting-based representation provides a tractable object for randomized smoothing.Source: A. Routing Containment Certification · p. 4
  • Randomly masking input positions yields per-expert smoothed routing probabilities used to select the smoothed Top-K experts.The mask retains a fixed number of tokens sampled uniformly from the input positions.Source: A. Routing Containment Certification · p. 4
  • ρ-relaxed containment allows perturbed Top-K experts to change, provided they remain within a larger certified pool of Z experts.This relaxes exact Top-K matching, which can be restrictive when experts are redundant or complementary and token routing varies with semantic roles.Source: A. Routing Containment Certification · p. 4
  • A sufficient margin condition on smoothed routing probabilities guarantees, with probability at least 1 − α, that perturbed Top-K experts remain inside the certified Top-Z pool.The condition applies to inputs within routing radius d_r; its bound depends on β and the probability Δ_r that masking intersects perturbed positions.Source: A. Routing Containment Certification · p. 5

B. Induced Sub-Network Certification

Given certified expert pools, CERTMOE restricts each layer’s routing candidates to those pools and applies a smoothing certifier to check prediction invariance.

B. Induced Sub-Network Certification

  • A base smoothing certifier verifies prediction invariance of the induced sub-network within its certified radius.The stated robustness condition gives g_sub(x′) = y with probability at least 1 − δ for inputs satisfying the condition.Source: B. Induced Sub-Network Certification · pp. 5–6
  • The induced sub-network keeps each original router but restricts its candidate experts to the certified pool at that layer.It recomputes routing scores and applies the original Top-K rule among the restricted candidates.Source: B. Induced Sub-Network Certification · p. 5
  • When routing containment holds at every layer, the induced sub-network and original MoE produce identical outputs.The original router’s selected experts already lie inside the certified pools under that condition.Source: B. Induced Sub-Network Certification · p. 6

C. MoE Robustness Certification

CERTMOE combines layer-wise routing-containment certificates with induced sub-network robustness to obtain a probabilistic robustness guarantee for the original MoE model.

C. MoE Robustness Certification

  • When routing containment and induced sub-network invariance both hold, the original MoE prediction remains y.The certified radius is bounded by the routing and sub-network radii, with a union bound limiting failure probability to Lα + δ.Source: C. MoE Robustness Certification · p. 6
  • The procedure certifies routing and the induced sub-network separately, then combines their radii into the final certified radius.Algorithm 1 implements the routing and sub-network stages with Monte Carlo sampling and confidence bounds.Source: C. MoE Robustness Certification · p. 6
  • CERTMOE permits perturbed inputs to activate different experts if their Top-K choices stay within the certified pool; otherwise, it abstains to preserve soundness.Routing variation within the pool is handled by induced sub-network certification.Source: C. MoE Robustness Certification · p. 6

V. EVALUATION

The evaluation is organized around certificate reliability, parameter sensitivity, failure sources, and computational overhead. The supplied material identifies a table of representative MoE LLMs but does not provide evaluation outcomes.

V. EVALUATION

  • The evaluation asks whether CERTMOE improves certificate reliability, how its parameters affect certification, what failure sources it reveals, and what overhead it incurs.The four questions cover reliability, sensitivity, failure attribution, and computational cost.Source: §V · pp. 6–7
  • Table II is scoped to representative MoE LLMs and defines “#Act./Total” as activated versus total parameters.For DeepSeek-V2-Lite, “64 + 2” denotes 64 routed experts plus 2 shared experts.Source: Table II · p. 7

A. Experimental Setup

Experiments evaluate smoothing-based certification on two text-classification benchmarks and three MoE LLMs, using discrete-token attacks and a fixed sampling configuration.

  • Experiments use SST-2 binary sentiment and AGNews four-class topic classification, evaluated on Qwen1.5-MoE-A2.7B, DeepSeek-v2-Lite, and Phi-3.5-MoE-instruct.The models span different depths, Top-K settings, and expert capacities.Source: A. Experimental Setup · p. 7
  • The comparison uses RanMASK and SelfDenoise, two text-domain smoothing methods, because the evaluated perturbations are defined over discrete tokens.SelfDenoise reconstructs noisy inputs before prediction using Qwen3-8B.Source: A. Experimental Setup · p. 7
  • Due to certification cost, evaluation samples 100 test inputs per dataset and uses 100 perturbed samples for prediction and 500 for each certification bound.The same sampling sizes apply to routing and sub-network certification; input mask ratio is 0.2.Source: A. Experimental Setup · p. 7
  • For CERTMOE, the held-out validation set selects ρ; the study evaluates certificates with TextBugger, DeepWordBug, and TextFooler under default TextAttack settings.The overall failure probability is 0.10, split as Lα = 0.05 and δ = 0.05.Source: A. Experimental Setup · p. 7

B. Evaluation Metrics

The evaluation separates certificate coverage from reliability: coverage counts inputs receiving certificates, while attack-observed violations and a binomial test assess reliability.

  • Certified ratio (CR) is the fraction of test inputs receiving a non-trivial certificate, with certified radius set to zero when certification fails.The table caption cautions that CR should not be used alone as a robustness indicator.Source: B. Evaluation Metrics · p. 7, Table III · p. 8
  • Violation rate (VR) is the fraction of certified inputs for which a counterexample exists within the certified region.Because exhaustive search is intractable, the study estimates VR using three attacks, each generating T adversarial candidates.Source: B. Evaluation Metrics · p. 7
  • The reported attack-observed VR aggregates counterexamples found by the attacks, which explore only a subset of each certified region.All attacks share a budget of d_i word modifications; character-level edits count as one unit per perturbed word.Source: B. Evaluation Metrics · p. 7
  • A one-sided exact binomial test evaluates whether observed VR provides significant evidence that the true violation probability exceeds 10%.A larger p-value indicates insufficient evidence to reject the null hypothesis that the probability is at most 0.1.Source: B. Evaluation Metrics · pp. 7–8

C. Certification Reliability (RQ1)

Baseline smoothing methods often issue certificates that attacks can violate, whereas CERTMOE issues fewer certificates under its hierarchical conditions and reports more reliable outcomes.

  • Across three MoE models, RanMASK and SelfDenoise often have relatively high certified ratios but substantial violation rates, especially on SST-2, where VR frequently exceeds 30%.CERTMOE-enhanced methods issue fewer certificates but produce empirically more reliable outcomes.Source: C. Certification Reliability (RQ1) · p. 8
  • Most methods show lower VR on AGNews than SST-2, while some SelfDenoise model–dataset pairs retain statistically significant violations after CERTMOE.The paper suggests subtle sentiment-changing substitutions and SelfDenoise input rewrites as possible explanations, and notes that denoising-based certifiers require additional treatment.Source: C. Certification Reliability (RQ1) · p. 8
  • On representative SST-2 settings, Fig. 4 reports empirical violation rates across perturbation budgets: RanMASK and SelfDenoise remain consistently high, while CERTMOE substantially reduces violations on DeepSeek-V2-Lite and Qwen1.5-MoE-A2.7B.Violation rates are measured as the fraction of perturbed tokens relative to sentence length.Source: C. Certification Reliability (RQ1) · p. 8, Figure 4 · p. 8

D. Parameter Sensitivity (RQ2)

The sensitivity analyses show that certified expert-pool size and mask settings affect coverage and reliability; failure attribution separates routing abstention from sub-network abstention.

D. Parameter Sensitivity (RQ2)

  • Larger certified expert pools make routing containment easier and certify more inputs, but overly permissive pools have higher violation rates than ρ = 0.2.The strict setting Z = K certifies no inputs; ρ trades containment permissiveness against certificate reliability.Source: D. Parameter Sensitivity (RQ2) · p. 8, Table IV · p. 9
  • Across tested mask ratios, RanMASK retains substantial violation rates, especially under TextFooler, while CERTMOE selectively certifies fewer inputs and substantially reduces VR.The results remain consistent across the evaluated (m_o, m_r) settings.Source: D. Parameter Sensitivity (RQ2) · p. 9, Table V · p. 9

E. Failure Attribution (RQ3)

  • Routing-containment abstention rises with model size and is amplified under SelfDenoise, whose input rewrites can shift expert selections.Routing containment is checked first; only samples that pass proceed to sub-network certification.Source: E. Failure Attribution (RQ3) · p. 9
  • On five Qwen1.5-MoE-A2.7B examples, RanMASK certificates were all falsified by adversarial substitutions, while CERTMOE abstained on three and preserved two supported certificates.CERTMOE attributes uncertified outcomes to routing containment or the induced sub-network.Source: E. Failure Attribution (RQ3) · p. 9

F. Computation Cost (RQ4)

CERTMOE roughly doubles certification runtime, with routing-containment and sub-network certification accounting for most of the cost. The discussion bounds the evaluation and identifies routing-aware certified training as future work, while the conclusion reports more reliable certificates.

F. Computation Cost (RQ4)

  • CERTMOE roughly doubles total runtime compared with RanMASK alone across most settings.The figure breaks down per-sample prediction and certification costs on SST-2 and AGNews across three MoE LLMs.Source: F. Computation Cost (RQ4) · p. 9, Figure 6 · p. 10
  • Routing-containment and sub-network certification make up the primary cost, with their relative shares varying across models and datasets.Routing-containment certification is larger in most settings; sub-network certification dominates for Qwen on AGNews, while Phi has relatively low sub-network cost because more samples are rejected at routing containment.Source: F. Computation Cost (RQ4) · p. 10
  • The runtime overhead is incurred only when a certificate is requested and does not affect standard inference.Additional persistent memory is O(LN + LZ), reducing to O(LN) because Z ≤ N.Source: F. Computation Cost (RQ4) · p. 10

VI. DISCUSSION

  • The evaluation is limited to text classification and representative MoE LLMs, so generalization to other architectures or generation tasks requires further study.The reported robustness metrics are tied to the considered perturbation model; runtime also depends on hardware and implementation.Source: §VI · p. 10

VII. CONCLUSION

  • The conclusion reports that CERTMOE consistently produces more reliable certificates on state-of-the-art MoE LLMs and benchmark datasets.The authors leave the design of routing-aware certified-training methods to future work.Source: §VII · p. 11, §VI · p. 11

Related papers

Loading 2610.29395v1…