RAPTOR: Role-Aware Private Training for Mixture-of-Experts

1 Introduction

• RAPTOR uses alternating shared and expert streams, expert-specific clipping and noise, a public expected-owner denominator, and a count-independent update schedule.The expert denominator is De = B/E; the schedule updates every expert at every step without conditioning on realized counts.Source: §1 · p. 2
• The paper identifies and formally characterizes clipping interference, update-scale dilution, and load-dependent signal-to-noise degradation in monolithic DP-MoE training.These mismatches arise because shared components see every record while experts see only routed records.Source: §1 · pp. 1–2
• Parallel composition over routing-induced owner groups makes updating all E experts within a layer cost no more in privacy terms than updating one.Shared and expert releases compose sequentially across training.Source: §1 · p. 2
• A bias bound tied to routing imbalance motivates selecting the layer with highest routing entropy on a small public corpus.The paper also reports that this bias is stable under bounded representation drift via a Voronoi abstraction.Source: §1 · p. 2
• Up to +3.24 points over monolithic and matched-scope DP baselines on Switch Transformer, with consistent improvements on OLMoE and DeepSeek-VL2-Tiny.The Switch Transformer evaluation covers four GLUE tasks at ε ∈{1, 4, 8}.Source: §1 · p. 2

2 Related Work

• Dense-model DP fine-tuning uses one clipping bound, noise scale, and denominator, although MoE shared components see every record and experts see only routed subsets.The passage names full-parameter fine-tuning and LoRA as private fine-tuning settings using this uniform treatment.Source: §2 · p. 3
• Batch-level load-balancing losses yield per-record gradients reflecting other records’ routing decisions and depend on private realized expert counts.The passage says these gradients break the per-record sensitivity required by DP-SGD.Source: §2 · p. 3
• The prior private MoE method treats the model as a single dense block, leaving shared, routing, and expert parameters indistinguishable.The passage says this leaves expert gradients suppressed, sparse updates diluted, and low-load experts with concentrated noise.Source: §2 · p. 3

3.1 Setup and Notation

• A sparse MoE has E experts per sparse layer, with shared experts seeing every token and routed experts seeing only assigned tokens.The setup distinguishes shared-stream and expert-stream quantities such as clipping bounds, noise multipliers, and learning rates.Source: §3.1 · p. 3
• Poisson subsampling gives a random private batch size, so updates normalize by the fixed public expected size B rather than the realized count.For gradient g and radius C, clipping scales g by min{1, C/∥g∥2}.Source: §3.1 · p. 3

3.2 Differential Privacy

• DP-SGD privatizes each step with per-record clipping and Gaussian noise, while Adam on released gradients is post-processing with no additional privacy cost.The passage describes privacy cost as the T-fold composition already charged for the released sequence.Source: §3.2 · pp. 3–4

3.3 Sparse MoE Models and Load Balancing

• The method assigns each record to one owner expert per layer and freezes the router, omitting auxiliary load-balancing losses that depend on private batch routing statistics.For top-k routing, the owner is the expert receiving the most of a record’s tokens across its assignments.Source: §3.3 · p. 4

4 Diagnosing Monolithic Private Optimization Failures

• Monolithic private optimization uses one clipping bound, noise scale, and denominator for parameters with different record exposure, producing three mechanism-level failures.The paper reports that empirical role-mismatch and per-expert SNR diagnostics directionally corroborate the analysis.Source: §4 · p. 4

4.1 Clipping Interference

• A single clipping factor dominated by dense blocks suppresses expert gradients even when the expert component alone is within the clipping bound.Increasing the shared bound enough for experts also inflates dense-block noise, motivating a separate expert clipping bound.Source: §4.1 · p. 4

4.2 Update-Scale Dilution

• Normalizing by B when only m_t,l,e records contribute rescales expert gradients toward zero by m_t,l,e/B.Using the realized expert count for normalization would expose private routing information; the public denominator De = B/E is unbiased under uniform routing.Source: §4.2 · p. 4

4.3 Load-Dependent Signal Degradation

• Fixed privacy noise per expert concentrates noise on low-load experts because useful signal scales with m_t,l,e.Routing entropy and its deficit from log E are linked to imbalance bias in the public-denominator estimator and motivate a candidate privacy-free layer-selection diagnostic.Source: §4.3 · p. 4

5.1 Public Layer Selection

• Layer selection requires a disjoint, format-matched public corpus; without one, the method defaults to the last sparse layer.The frozen router is applied to the public corpus to estimate each candidate layer’s owner distribution; the selection consumes no privacy budget.Source: §5.1 · p. 5

5.2 Record-Level Expert Ownership

• A deterministic gate-weighted plurality rule assigns each record to exactly one owner expert, partitioning private records into disjoint expert groups.Ties go to the lowest index; the forward pass may still use top-k routing, while only the owner expert receives the record’s expert gradient.Source: §5.2 · p. 5

5.3 Shared and Expert Stream Updates

• Expert updates use expert-specific clipping and the public denominator D_e = B/E, so realized owner counts are not revealed and sensitivity is C_e/D_e.The shared stream uses standard per-record clipping and noise; the expert loss weights records by the shared stream’s predicted error probability before clipping.Source: §5.3 · pp. 5–6

5.4 Alternating Optimization and Layer Scope

• Each outer cycle draws independent Poisson subsamples, then updates shared parameters followed by all experts; the method scopes private training to one selected layer.Alternation lets shared representations and expert residual corrections co-adapt, with each stream affecting the residual weights used by the other.Source: §5.4 · p. 6

6 Privacy Analysis

• The privacy analysis assumes add/remove adjacency and releases only the final model, not owner assignments, expert counts, residual weights, or intermediate gradients.These are the stated release conditions for the analysis.Source: §6 · p. 6

6.1 Parallel Composition Across Experts

• Because each record belongs to exactly one owner group and each expert mechanism depends only on its group and public quantities, all E expert mechanisms compose in parallel at the layer’s maximum privacy cost.The owner map must be deterministic and fixed for the stage; the joint mechanism is (max_e ε_l,e, max_e δ_l,e)-DP when each group mechanism has its corresponding guarantee.Source: §6.1 · p. 6

6.2 Composition Across Streams and Layers

• Shared updates and trained layers compose sequentially, while expert mechanisms within each layer contribute one layer-level privacy mechanism rather than an E-fold convolution.Fresh independent Poisson subsamples support adaptive PRV composition across interleaved shared and expert updates.Source: §6.2 · p. 6

7.1 Bias-Variance Decomposition and Layer Selection

• The public-denominator estimator’s squared bias vanishes under uniform routing and grows quadratically with routing imbalance, while its variance separates sampling error from DP cost.The estimator’s decomposition is given under Poisson subsampling; the displayed variance expression is not reproduced in the supplied passages.Source: RAPTOR · p. 7

RAPTOR

• Maximizing routing entropy on public data minimizes the guaranteed aggregate bias bound and gives a privacy-free criterion for selecting the protected layer.The selection uses only the public corpus and frozen router; it is a bias-based heuristic, not a guarantee of improved DP-Adam utility or per-expert signal-to-noise ratio.Source: RAPTOR · pp. 7–8

7.2 Stability of Expert Ownership

• Under the Voronoi abstraction and a boundary-mass condition, representation drift changes ownership only for records near a boundary, with reassignment probability and routing-mass drift scaling linearly with drift.The ownership map is frozen before training; the informal stability result bounds which records may change owner as representations move.Source: §7.2 · p. 8

8.1 Setup

• The evaluation covers four GLUE tasks on Switch, OLMoE with top-8 routing, and DeepSeek-VL2-Tiny on ScienceQA; accuracy is reported on development sets.The private methods use ε ∈ {1, 4, 8}; baselines include non-private LoRA and full fine-tuning, plus two monolithic DP methods.Source: §8.1 · p. 8

8.2 Results on Switch Transformer

• +0.23 to +3.24 points at ε = 8 and +1.51 to +2.92 at ε = 1 over Monolithic DP-Adam LoRA; RAPTOR has the highest accuracy in all twelve Switch task–budget cells.The advantage grows at tighter budgets on SST-2 and QNLI, but not on MNLI or QQP.Source: §8.2 · p. 8, Table 1 · p. 9
• Entropy-selected layers match or exceed first- and last-sparse-layer variants in every cell, including non-endpoint selections for MNLI and QQP.The selected blocks are 7 for MNLI and 9 for QQP.Source: §8.2 · p. 9

8.3 Results on OLMoE

• +2.63 (SST-2), +0.84 (MNLI), +4.50 (QNLI), and +0.70 (QQP) are RAPTOR’s ε = 1 gains over Monolithic DP-Adam LoRA on OLMoE.RAPTOR has the highest accuracy in all twelve task–budget cells; at ε = 8, the corresponding gains are +0.69, +2.18, +1.74, and +0.32.Source: Table 2 · p. 9, §8.3 · p. 9

8.4 Empirical Privacy Auditing via Membership Inference

• The audit finds sharply lower LiRA TPR for all DP methods than non-private training, while RAPTOR has the highest TPR/AUC among DP methods on LiRA and loss-attack AUC at every ε.It ties Monolithic DP-LoRA on loss-attack TPR at ε = 1; the comparison is based on a single run per cell.Source: §8.4 · p. 9

• Monolithic DP treats sparse MoEs as dense blocks, mismatching their shared and expert structure.Source: §9 · p. 9
• RAPTOR addresses the mismatch with alternating streams, expert-specific clipping and noise, public normalization, entropy-based layer selection, and a parallel-composition DP guarantee.Source: §9 · p. 9
• RAPTOR’s gains hold across Switch, OLMoE, and DeepSeek-VL2-Tiny, and are typically largest at tight privacy budgets.Source: §9 · p. 9

• A frozen router precludes privately updating routing parameters within a stage; adapting it would require re-accounting for ownership changes.Source: Limitations · p. 9
• The public denominator De = B/E is privacy-safe but not oracle-optimal, and can over- or under-scale expert updates under heavy imbalance.Source: Limitations · p. 9
• Small owner groups may have clipped sums dominated by high-weight records, risking fit to label noise; the authors did not run label-noise or subgroup diagnostics.Clipping after residual weighting preserves the stated sensitivity bound and DP guarantee regardless of the weights.Source: Limitations · p. 10
• Future work includes private router adaptation, adaptive normalization, and extensions to generative and larger-scale multimodal MoEs.Source: Limitations · p. 10

A.1 Structured Private Optimization

• Unlike dense layer- or client-wise group clipping (Thakkar et al., 2019), RAPTOR aligns clipping, normalization, and composition with routing-induced shared and expert roles.Subspace, low-rank, and noise-shaping methods are described as orthogonal to this framework.Source: §A.1 · p. 15

A.2 Partition-Based DP Composition

• Parallel composition applies across disjoint owner groups because the frozen public router determines ownership and the owner map is never released; changing partitions across layers are handled by sequential composition.The owner-map computation has zero privacy cost because it is internal bookkeeping derived from the public router and private inputs.Source: §A.2 · p. 15

A.3 Sparse Mixture-of-Expert Models

• Private batch routing statistics can break DP-SGD sensitivity, so RAPTOR freezes the router, drops the auxiliary load-balancing loss, and normalizes expert updates by the public denominator De = B/E.Experts active on every record enter the shared stream, while conditionally routed experts enter the expert stream.Source: §A.3 · pp. 15–16

B.1 Public and Private Quantities

• The pretrained checkpoint, frozen router, schedule, hyperparameters, and public corpus are public; owner maps and realized counts remain internal, and each record affects exactly one owner group.If the router or checkpoint were trained on private data without DP, that step would incur additional privacy cost; per-update sensitivity is independent of realized expert count.Source: §B.1 · p. 17, §B.2 · p. 17

B.3 Proof of Lemma 1

• Alternating updates preserve parallel composition because model states are post-processing of prior privatized gradients, expert steps depend only on their owner subsets and public quantities, and the schedule does not branch on private counts.Empty owner subsets receive noise-only updates rather than being skipped.Source: §B.3 · pp. 17–18

B.4 Proof of Lemma 2

• Residual weighting does not add privacy cost: its cross-group influence is mediated by the already-private model state, and clipping after weighting keeps sensitivity and Gaussian-noise calibration unchanged.Weights are used only inside clipping and are neither released nor used in control flow; standard sequential composition applies to the fixed-parameter steps.Source: §B.4 · pp. 18–19

B.6 Proof of Theorem 2

• The public-denominator error bound is controlled by routing imbalance, with a simpler bound proportional to χ2(q∥u) supporting the public-data entropy criterion.In the surrogate SGD analysis, the imbalance contribution vanishes under uniform routing; this is not a convergence guarantee for DP-Adam on non-convex MoE networks.Source: §B.6 · p. 20, §B.7 · p. 20

B.8 Scaling of the DP Term

• Under balanced routing, the DP term matches the standard per-expert DP mean-estimation rate up to logarithmic factors when B = Θ(n), while ownership stability is bounded by representation drift and boundary regularity.The stability bound is P(ãC(X) ≠ aC(X)) ≤ 2κΔCr and ∥q̃ − q∥1 ≤ 4κΔCr.Source: §B.8 · p. 20, §B.9 · p. 21

B.10 Voronoi-aware expert estimator Error

• The expert-estimator error is decomposed into variance and squared bias relative to prototype gradient ν_e, with the bias bound containing routing imbalance and prototype mismatch terms.The displayed bound includes 2(Eq_e − 1)^2∥μ_e∥_2^2 + 2∥μ_e − ν_e∥_2^2; the DP noise is independent of the sampling term.Source: §B.10 · p. 24

B.11 Scope of the Bias-SNR Connection (Remark 2 details)

• The bias measure β_e is symmetric around q_e = 1/E and measures deviation from a normalization convention, not training quality; aggregate routing entropy cannot bound an individual expert’s inverse SNR.A starved expert can have vanishing SNR while routing entropy remains near-maximal, so no finite function of entropy bounds 1/SNR_e for all valid routing distributions.Source: §B.11 · p. 25

C Experiment Details

• Experiments tune each method independently on SST-2 using 100 trials, then freeze its best configuration for MNLI, QNLI, and QQP, distinguishing in-task tuning from cross-task transfer.Trials use Poisson sampling and fixed expected denominators B for the shared stream and D_e = B/E for the expert stream, independent of realized batch size.Source: C Experiment Details · p. 26

D More Expriments on Other Models

• At ε = 1, RAPTOR reaches 86.98% on ScienceQA versus 85.12% for Monolithic DP-Adam LoRA (+1.86); it ties at ε = 4 and slightly exceeds non-private LoRA at ε = 8 in a single-run artifact.This DeepSeek-VL2-Tiny vision-language test protects the last routed-expert layer by default because no disjoint, format-matched public corpus was available; the result does not establish private training exceeding its non-private upper bound.Source: D More Expriments on Other Models · p. 27, Figure 3 · p. 27

E.1 Component Ablation

• RAPTOR’s full component set gains 0.23–3.24 pts over Monolithic DP-Adam LoRA; alternation contributes most, up to 1.97 pts on QNLI at ε ≈8 and 2.09 pts on QQP at ε = 1.The public denominator contributes 0.05–0.98 pts, expert-specific clipping 0.06–0.62 pts, and the residual objective 0.04–0.66 pts; high-noise amplification varies by task.Source: §E.1 · p. 28, Table 5 · p. 28

E.2 Matched-Scope Diagnostics

• With layer, scope, budget, and steps matched, Baseline A trails RAPTOR by 0.36–3.85 points on MNLI, QNLI, and QQP at ε = 8, while expert-only training collapses toward random accuracy.The expert-only baseline reaches 32–36% on MNLI and ≈50% on QNLI; shared-only training is competitive on SST-2 but trails on QNLI and QQP at every budget.Source: §E.2 · p. 29, Table 6 · p. 29

E.3 Sensitivity to the Shared/Expert Budget Split ρ

• At ρ = 0.95, SST-2 accuracy drops from 93.46 to 92.09 as ε rises from 4 to 8, versus 94.15 for ρ = 0.9 at ε = 8; RAPTOR uses ρ = 0.9 by default.Sensitivity to the shared/expert split is greatest at ε = 1 and shrinks at larger budgets; the most shared-dominant tested split can under-budget expert learning.Source: §E.3 · p. 29, Figure 4 · p. 30

F Empirical Routing Entropy Analysis

• H/log E is 0.972 for SST-2 block 1, 0.682 for MNLI block 7, 0.902 for QNLI block 11, and 0.837 for QQP block 9, each task’s most balanced layer.The corresponding selected-layer χ2 values are 0.118, 1.262, 0.393, and 0.683, respectively.Source: F Empirical Routing Entropy Analysis · p. 31
• Entropy selects blocks 1, 7, 11, and 9 for SST-2, MNLI, QNLI, and QQP, respectively, with at least 0.029 nats over each runner-up.The fixed last-layer heuristic misses the optimum for SST-2 by 0.311 nats and also misses QQP.Source: F Empirical Routing Entropy Analysis · p. 31
• All four tasks select the correct layer at n = 1,000 calibration examples, while stability ranges from n = 10 for SST-2 to n ≥1,000 for QQP.The section recommends n ≥1,000 as a conservative default; when the entropy margin exceeds 0.05 nats, more examples do not change the ranking or selected block.Source: F Empirical Routing Entropy Analysis · p. 31
• Public-to-development entropy transfer is close for QNLI and looser for SST-2 and QQP, but all six MNLI layers show a systematic gap above the diagonal.Entropy-selected layers sit closer to the diagonal than collapsed, non-selected layers; MNLI’s top two public-corpus layers differ by only 0.008 nats, so their development-set ranking may differ.Source: F Empirical Routing Entropy Analysis · p. 31, Figure 7 · p. 34

G Mechanism Diagnostics

• RAPTOR reduces role mismatch versus full Monolithic DP-Adam from 0.591 to 0.401 on SST-2, 0.560 to 0.337 on MNLI, and 0.619 to 0.397 on QNLI.It does not uniformly minimize this diagnostic relative to Monolithic DP-Adam LoRA, though it avoids the pronounced mismatch of full monolithic DP-Adam.Source: G Mechanism Diagnostics · p. 35
• RAPTOR reduces SNR-CV versus Monolithic DP-Adam LoRA on SST-2, MNLI, and QNLI, but not on QQP; low SNR-CV can reflect uniformly weak updates.Full Monolithic DP-Adam has lower SNR-CV on SST-2, MNLI, and QNLI, where its mean expert SNR is far lower than RAPTOR’s.Source: G Mechanism Diagnostics · p. 36
• The diagnostics are indicative rather than conclusive because only a limited number of configurations were completed, but trends show reduced role mismatch and increased mean expert SNR on harder GLUE tasks.The reported mechanism trends are directionally consistent with downstream accuracy gains and suggest RAPTOR prevents expert updates from being suppressed or diluted by a dense monolithic optimizer.Source: G Mechanism Diagnostics · pp. 35–36
• For every task, the entropy-selected layer has lower d CV_l and higher bH_l than the most-collapsed layer, consistent with higher routing entropy accompanying more uniform load.These corpus-level statistics are computed on public calibration corpora before private training, and are identical across private training methods because the router is frozen.Source: G Mechanism Diagnostics · p. 36, Table 9 · p. 36

H Budget Allocation: Surrogate Analysis and Empirical Sensitivity

• ρ* increases with shared-stream sensitivity relative to expert sensitivity and decreases as the number of experts E grows, because expert-stream noise is amplified by E².The shared-dominant choice is attributed to sensitivity asymmetry, not to expert count.Source: H Budget Allocation: Surrogate… · p. 37
• λe/λs ≲ 6.3 × 10−3 indicates greater shared-stream sensitivity, but the bound needed to certify ρ* ≥0.9 is not met, so ρ = 0.9 is empirically validated rather than certified optimal.The required ratio is λe/λs ≤2.1 × 10−5 for E = 8 and Cs = Ce.Source: H Budget Allocation: Surrogate… · p. 37
• Tab. 10 reports clear shared-stream sensitivity (λs = 2.305 × 10−5) and negligible expert sensitivity; the signed expert estimate’s magnitude is used conservatively as an upper bound.Source: H Budget Allocation: Surrogate… · p. 37

I Runtime and Memory

• RAPTOR trains 17.30M parameters (5.62% of the model), comparable in scope to LoRA’s 14.16M (4.60%) and far below full-model training at 307.85M.The reported accuracy gains over Monolithic DP-Adam are therefore not explained by training more of the model.Source: I Runtime and Memory · p. 39
• 112.16GB peak memory and 4.51s per step are reported for RAPTOR, versus 130.74GB and 4.07s for LoRA and 165.51GB and 7.23s for full-model training.RAPTOR’s total compute is 1.65 GPU-hrs, about 37% below the full-model baseline; the comparison is implementation-level and differs in training scope.Source: I Runtime and Memory · p. 39, Table 11 · p. 39