PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust CurveMoE: Multi-Norm Adversarial Defense for Mixture-of-Experts Models via Mode Connectivity

Xu Zhang, Ren Wang

arXiv:2608.26043v1cs.LG

TL;DR

Existing multi-norm defenses force competing robustness objectives into one parameter configuration, increasing cost and creating unfavorable trade-offs. Robust CurveMoE instead derives complementary experts from a low-loss connectivity curve, selectively expertizes influential layers, and uses sparse routing. On CIFAR-100 and ImageNet-100, it consistently improves clean, norm-specific, and Union accuracy over MSD and ERMC.

  • Problem

    Existing multi-norm defenses optimize competing norm-specific objectives within a single parameter configuration, causing robustness trade-offs and substantial training overhead.

  • Method

    Robust CurveMoE connects norm-specialized models through a robust connectivity curve, derives constrained experts, selectively expertizes influential layers, and reduces curve cost through contribution-guided partial updating.

  • Results

    Robust CurveMoE consistently improves clean, norm-specific, and Union accuracy over MSD and ERMC on CIFAR-100 and ImageNet-100.

  • Takeaways & Limitations

    The framework provides an efficient MoE-based approach to multi-norm adversarial defense by exploiting complementary curve-derived robustness profiles with sparse routing.

Abstract

from arXiv · show

Multi-norm adversarial defense aims to protect neural networks against perturbations defined by different norm constraints, but existing methods typically optimize competing robustness objectives within a single parameter configuration, leading to substantial training cost and unfavorable robustness trade-offs. We propose Robust CurveMoE, an efficient mixture-of-experts framework that connects models specialized for different perturbation norms through a low-loss path and exploits the complementary robustness profiles of models along this path. Robust CurveMoE derives clean and norm-specialized experts from robustness-constrained curve locations and selectively expertizes only influential layers, while sharing the remaining parameters across routing paths. To further reduce curve-construction cost, we introduce contribution-guided partial updating, which selects influential curve parameters using initialization-based gradient scores. We also theoretically bound the objective gap between partial and full curve optimization. Experiments on CIFAR-100 and ImageNet-100 with WideResNet and Vision Transformer architectures show that Robust CurveMoE consistently improves clean, norm-specific, and Union accuracy over MSD and ERMC. In particular, it improves Union accuracy by 2.37 and 2.13 percentage points over the strongest baseline on CIFAR-100 and ImageNet-100, respectively. Extensive ablations further validate the effectiveness of partial updating, selective expertization, and robustness-constrained expert selection.

I. INTRODUCTION

Multi-norm defenses face competing robustness objectives, clean-accuracy trade-offs, and high adversarial training cost. Robust CurveMoE addresses these challenges by converting complementary models along a robust connectivity curve into selectively expertized, sparsely routed experts.

  • Motivation: Multi-norm defenses optimize competing objectives in one parameter configuration, so improving one norm can compromise another or clean accuracy.
  • Motivation: Multiple threat models also increase training overhead because adversarial examples must be generated under several norm constraints.
  • Motivation: Robust mode connectivity supplies curve models with distinct, complementary robustness profiles without independently training every expert from scratch.
  • Contributions: Only influential layers are expertized while remaining parameters are shared, reducing duplication and preserving efficient conditional computation.
  • Contributions: Robust CurveMoE converts curve-derived models into specialized experts and uses input-dependent sparse routing to exploit their complementary behaviors.
  • Contributions: Contribution-guided partial updating reduces curve-construction cost by selecting influential parameters before training, with a theoretical bound on the objective gap from full optimization.

III. ROBUST CURVEMOE

Robust CurveMoE builds a structured expert pool from a low-loss curve connecting norm-specialized models, then integrates selected experts into a sparse MoE. Shared layers preserve efficiency while top-1 routing activates a suitable expert for each input.

  • Robust CurveMoE: Robust CurveMoE samples curve models to construct a structured pool of experts with complementary robustness profiles.
  • Robust CurveMoE: Only layers most influential to multi-norm robustness receive expert-specific parameters, while the remaining layers are shared.
  • Multi-norm defense: Adversarial perturbations are constrained by ℓp threat regions, and multi-norm defense seeks robustness over the union of several threat models.
  • Mixture-of-experts: The router produces scores for K experts and top-1 routing activates only the highest-scoring expert for each input.
  • Mode connectivity: The framework connects independently trained endpoint models with a continuous low-loss curve whose locations can exhibit complementary robustness profiles.

B. Curve-Derived Expert Pool for Multi-Norm MoE

Robust CurveMoE builds complementary multi-norm experts from a robust connectivity curve, then preserves efficiency through robustness-constrained selection and selective expertization. Contribution-guided updating further reduces curve-construction cost while theoretically controlling the objective gap.

  • Curve-derived expert pool: Candidate experts are sampled from a single robust connectivity curve connecting norm-specialized models, avoiding independently training complete experts for each norm.The curve connects models trained against ℓ1 and ℓ∞ perturbations and is optimized across ℓ1, ℓ2, and ℓ∞ threats.
  • Curve-derived expert pool: Curve locations exhibit distinct robustness preferences: ℓ1 robustness is stronger near the ℓ1 endpoint, ℓ∞ robustness near the ℓ∞ endpoint, and ℓ2 robustness at intermediate peaks.These observations motivate selecting complementary curve models rather than treating all locations as equivalent.
  • Curve-derived expert pool: MoE routing can preserve specialized robustness by selecting an appropriate expert per input, whereas aggregating all curve models may dilute a specialist’s contribution under its corresponding attack.The contrast is between input-adaptive sparse routing and prediction aggregation across models with differing robustness preferences.
  • Selective expertization: Selective expertization instantiates expert-specific parameters only in adversarially sensitive layers and shares the remaining backbone parameters across routing paths.Layer sensitivity is measured across candidate MoE layers, with ViT candidate layers corresponding to MLP modules in Transformer blocks.
  • Sensitivity-guided design: Layer contributions are highly non-uniform and partly objective-dependent, supporting targeted expertization and parameter updating rather than replicating or optimizing every layer.Some Transformer blocks are influential across objectives, while others differ between clean and adversarial objectives.
  • Robustness-constrained selection: The candidate set retains curve locations whose overall robust accuracy is within tolerance τcand of the best sampled model, excluding severely imbalanced specialists.Smaller τcand enforces stricter multi-norm robustness, while larger values admit broader specialization.
  • Partial curve updating: Contribution-guided partial updating selects influential curve parameters using initialization-based gradient scores, while freezing the remainder to reduce construction cost.The score Gm serves as a practical proxy for the parameter displacement excluded by restricted optimization.
  • Partial curve updating: Theorem 1 bounds the objective gap between full and restricted curve optimization under smoothness and stationarity assumptions.The bound motivates retaining parameters expected to undergo substantial adaptation.

A. Experimental Setup

Experiments evaluate Robust CurveMoE across two datasets, two architecture families, three adversarial threat models, and clean accuracy. The setup follows endpoint training, curve construction, expert selection, and MoE fine-tuning, with fixed default hyperparameters.

  • Datasets and architectures: Experiments use CIFAR-100 with WideResNet-28-10 and ImageNet-100 with ViT-Tiny/16 to test generality across convolutional and Transformer architectures.The two settings cover different model families and datasets.
  • Threat models and evaluation: Evaluation considers ℓ1, ℓ2, and ℓ∞ perturbations, using AutoAttack for separate robust-accuracy measurements and clean test samples for standard accuracy.The perturbation budgets differ between CIFAR-100 and ImageNet-100.
  • Training settings: Training proceeds from ℓ1- and ℓ∞-robust endpoints to robust curve training and subsequent MoE fine-tuning.On CIFAR-100, endpoint, curve, and MoE stages use 150, 50, and 50 epochs, respectively; ImageNet-100 uses 30, 20, and 20 epochs.
  • Training and construction procedure: Algorithm 1 defines threat models P = {1, 2, ∞}, evaluation objectives Q = {std, ℓ1, ℓ2, ℓ∞}, updating ratio ρ, MoE layer count K, candidate tolerance τcand, and fine-tuning weight α.The procedure initializes endpoints and the curve, updates only selected parameters, samples curve locations, evaluates objectives, and constructs the MoE.
  • Training and construction procedure: The construction procedure selects top-contributing MoE layers, builds a shared backbone, instantiates experts, initializes routers, freezes shared parameters, and fine-tunes experts and routers.This implements selective expertization after curve evaluation and candidate selection.
  • Baselines and hyperparameters: Default comparisons use MSD and ERMC with ρ = 30%, K = 3, τcand = 2 percentage points, and α = 0.6.These settings control partial updating, selective expertization, candidate robustness range, and objective balancing.

B. Overall Multi-Norm Robustness

Robust CurveMoE achieves the strongest clean and multi-norm adversarial performance on CIFAR-100 and ImageNet-100. Its Union accuracy improves over ERMC and MSD while maintaining higher norm-specific robustness.

  • Union accuracy requires correct classification under ℓ1, ℓ2, and ℓ∞ threat models, making it a stricter multi-norm robustness measure.
  • Union accuracy rises to 18.23% on CIFAR-100, compared with 15.86% for ERMC and 15.21% for MSD.
  • Union accuracy reaches 39.65% on ImageNet-100, exceeding ERMC at 37.52% and MSD at 35.84%.
  • Robust CurveMoE achieves the best performance across all reported evaluation metrics on both datasets.

C. Robustness of Expert Selection

The candidate tolerance controls the trade-off between admitting specialized experts and limiting their cross-norm vulnerability. Larger tolerances improve standard evaluations but weaken robustness to targeted routing-and-classification attacks.

  • The targeted attack jointly encourages routing to the ℓ1-specialized expert and misclassification by that expert.
  • Increasing τcand generally preserves or improves clean and norm-specific adversarial accuracy.
  • Targeted-expert accuracy falls from 14.58% to 9.74% on CIFAR-100 as τcand increases from 2 to 10.
  • Targeted-expert accuracy drops from 38.18% to 15.67% on ImageNet-100 as τcand increases from 2 to 10.
  • The results support using τcand = 2 to limit cross-norm-vulnerable experts.

D. Ablation Study

Ablations examine partial curve updating and the number of expertized layers. Partial updating closely matches full updating, while increasing MoE layers improves robustness initially but eventually yields diminishing returns and greater overhead.

  • Effect of partial updating ratio: Reducing the updating ratio to 30% or 50% has only a minor effect on clean and norm-specific curve performance across both datasets.
  • Effect of the number of MoE layers: Union accuracy increases from 18.23% to 19.45% on CIFAR-100 when MoE layers increase from K = 3 to K = 5.
  • Effect of the number of MoE layers: Union accuracy increases from 39.65% to 42.93% on ImageNet-100 when MoE layers increase from K = 3 to K = 5.
  • Effect of the number of MoE layers: Increasing K to 7 produces only marginal changes and slightly reduces Union accuracy on both datasets.
  • Effect of the number of MoE layers: The default K = 3 balances multi-norm robustness against the training and parameter overhead of additional expert-specific layers.

E. Effect of Warm-up and Load Balancing

Warm-up and load balancing do not provide consistent improvements across CIFAR-100 and ImageNet-100. Robust CurveMoE therefore omits both components for a simpler training procedure without sacrificing reported robustness.

  • Neither warm-up nor load balancing consistently improves performance across the two datasets.
  • On CIFAR-100, warm-up slightly improves Union accuracy, while load balancing provides only marginal gains on some norm-specific metrics.
  • On ImageNet-100, the configuration without warm-up or load balancing achieves the highest clean, ℓ1, and ℓ∞accuracy.
  • The method omits both components, simplifying training without sacrificing clean or multi-norm robustness.

V. CONCLUSION

Robust CurveMoE uses robust connectivity curves to derive complementary experts and sparse routing for efficient multi-norm adversarial defense. Experiments show consistent gains across clean, norm-specific, and Union accuracy, while ablations support its efficiency mechanisms.

  • Framework: Robust CurveMoE derives complementary norm-specific experts from a robust connectivity curve instead of independently training multiple complete experts.The framework exploits diverse robustness profiles along the curve through input-dependent sparse routing.
  • Efficiency: Selective expertization targets layers most sensitive to adversarial objectives, reducing parameter overhead while retaining strong robustness.
  • Efficiency: Contribution-guided partial curve updating reduces robust curve-construction cost by updating selected influential parameters.
  • Routing: Cross-norm robust candidate selection mitigates vulnerabilities from adversarially manipulated routing toward overly specialized experts.
  • Results: Experiments across CIFAR-100 and ImageNet-100 with convolutional and Transformer architectures consistently improve clean, norm-specific, and Union accuracy over representative baselines.Ablations also support partial updating and selective expertization under reduced training and parameter overhead.

APPENDIX PROOF OF THEOREM 1

The proof bounds the objective gap between full and restricted curve optimization by relating the projected full-update solution to the restricted optimum under smoothness and stationarity assumptions.

  • Restricted optimization: The proof defines the restricted update using the projection PS onto the selected parameter subspace S.
  • Smoothness bound: L-smoothness of J provides the inequality used to compare the objective at the projected and full optimization points.
  • Stationarity: Because ω∗ is stationary for full curve optimization, the proof eliminates the first-order term in the objective comparison.
  • Restricted optimum: The restricted optimum is compared with the projected full solution because the projected displacement is feasible for the restricted problem.
  • Conclusion: Combining the preceding inequalities yields the stated objective-gap bound and proves Eq. (17).
Loading 2608.26043v1…