Source-linked AI summary
Fairness-Aware Mixture-of-Experts via Subgroup Reweighting and Gate Regularization
Sunhee Hwang
TL;DR
The paper addresses fairness disparities caused by subgroup imbalance and routing-induced bias in Mixture-of-Experts models. It proposes FAMoE, combining subgroup reweighting with gate entropy regularization, and reports improved fairness while maintaining competitive predictive performance and more balanced, interpretable expert utilization.
Problem
Subgroup imbalance can cause MoE gating networks to correlate routing with sensitive attributes, while shared or decoupled fairness methods inadequately handle heterogeneous subgroups and end-to-end prediction alignment.
Method
FAMoE combines subgroup reweighting with gate entropy regularization to correct routing-induced bias end-to-end and prevent experts from being monopolized by specific subgroups.
Results
FAMoE consistently improves the fairness–accuracy trade-off across target and sensitive-attribute settings, while an ablation reduces EO from 13.3 to 4.2 with reweighting and to 4.1 after adding gate entropy regularization.
Takeaways & Limitations
Routing distributions can serve as interpretable diagnostics of subgroup behavior, while explicitly correcting imbalance and routing promotes more balanced expert assignment.
Abstract
from arXiv · showhide
Deep learning models often produce performance disparities across demographic groups, due to the training data imbalance with respect to sensitive attributes such as gender or age. To address this problem, existing work has explored fair representation learning, data re-sampling, and adversarial training, which can be broadly categorized into two main approaches. Single-stage methods typically learn a shared representation for fairness, but often struggle to handle heterogeneous subgroup distributions. Two-stage methods learn representations separately from the final prediction task, which can lead to misalignment between fairness objectives and downstream predictions. We identify routing-induced bias, a failure mode in which subgroup imbalance drives the gating network to route subgroups onto a few experts, and propose an end-to-end Mixture-of-Experts (MoE) framework that corrects it. Specifically, we apply subgroup reweighting to correct data imbalance, and introduce gate entropy regularization to prevent routing from collapsing onto subgroup attributes, keeping expert utilization both balanced and interpretable. Beyond improving fairness, the routing distribution offers an interpretable view of how subgroups are allocated across experts. Experimental results demonstrate that the proposed approach improves fairness while maintaining competitive predictive performance.
I. INTRODUCTION
The paper identifies routing-induced bias in MoE models under subgroup imbalance and proposes FAMoE to correct it end-to-end. FAMoE combines subgroup reweighting with gate entropy regularization, improving the fairness–accuracy trade-off while making expert utilization more balanced and interpretable.
- I. INTRODUCTION: Subgroup imbalance underfits rare intersections of target and sensitive attributes, producing the largest performance disparities for scarcest subgroups.Majority subgroups receive more training exposure and gradient signal than underrepresented subgroups.
- I. INTRODUCTION: Single-stage shared representations struggle with heterogeneous subgroup characteristics, while multi-stage pipelines can misalign fairness learning with downstream prediction.These limitations motivate separate computational pathways optimized jointly with prediction.
- I. INTRODUCTION: Routing-induced bias occurs when subgroup imbalance correlates gating decisions with sensitive attributes, concentrating subgroups on a few experts and skewing utilization.The failure mode is specific to conditional routing and is not captured by shared-representation fairness methods.
- I. INTRODUCTION: FAMoE combines subgroup reweighting with gate entropy regularization to prevent experts from being monopolized by specific subgroups.The framework corrects routing-induced bias end-to-end while preserving interpretable routing.
- I. INTRODUCTION: The routing distribution provides an interpretable diagnostic of subgroup allocation across experts, and experiments show consistent fairness–accuracy improvements with more balanced utilization.Results are reported across multiple target and sensitive-attribute settings on CelebA.
II. RELATED WORK
Mixture-of-Experts models provide input-dependent computation and can model heterogeneous data through expert specialization. However, how subgroup imbalance shapes routing decisions remains underexamined in fairness-aware MoE research.
- A. MoE and Routing Dynamics: MoE dynamically assigns inputs to experts, enabling input-dependent processing, increased capacity, and specialization across regions of the input space.Prior MoE work has emphasized efficiency, scalability, and selective expert activation.
- A. MoE and Routing Dynamics: Existing fairness-aware MoE research includes counterfactual expert-selection mechanisms but does not analyze the interaction between data imbalance and routing decisions.This leaves subgroup-driven routing behavior insufficiently examined.
B. Learning Fairness Representation
Visual-classification fairness methods mainly modify shared representations or training data to reduce sensitive information and imbalance effects. Their reliance on a globally shared feature space limits flexibility for heterogeneous subgroups, motivating conditional computation pathways such as MoE.
- II. RELATED WORK: Representation-learning methods use joint optimization, adversarial gradient reversal, or mutual-information objectives to reduce dependence between features and sensitive variables.These approaches enforce fairness constraints within end-to-end shared representations.
- II. RELATED WORK: Feature-space interventions suppress sensitive cues or disentangle target-specific, sensitive, and shared components within a common representation.MFD, Fair-VPT, and FD-VAE exemplify these strategies.
- II. RELATED WORK: Data-level methods address imbalance and spurious correlations through hard-sample mining, balanced training-pair construction, and group-aware correlation suppression.Examples include SHaSAM, FairCL, and Fair-GDMS.
- II. RELATED WORK: Shared-space methods restrict representational flexibility for heterogeneous subgroup distributions, indicating a need for input-dependent computation pathways.MoE offers this structural flexibility but remains underexplored for fairness.
III. FAIRNESS-AWARE MIXTURE-OF-EXPERTS FRAMEWORK
The FAMoE framework uses a shared backbone, multiple experts, and a gating network to produce input-dependent predictions. Expert outputs are combined using routing weights computed from the shared feature representation.
- B. Expert Prediction and Gated Aggregation: A shared backbone encoder maps each input image x to a feature representation z, which feeds both the expert networks and the gating network.The shared backbone supports controlled comparisons across MoE modeling choices.
- B. Expert Prediction and Gated Aggregation: Each expert independently processes z and produces an expert-specific logit through its classifier.The expert outputs provide distinct prediction pathways for the same shared feature.
- B. Expert Prediction and Gated Aggregation: The gating network computes routing weights for the experts, and the final prediction aggregates expert outputs using those weights.This enables adaptive combination of multiple predictions for diverse input patterns.
- B. Expert Prediction and Gated Aggregation: For binary classification, the aggregated logit is converted into a prediction probability with the sigmoid function.The sigmoid operates on the final aggregated output.
C. Subgroup Reweighting for Joint Attribute Imbalance
The method addresses joint imbalance across target and sensitive attributes by assigning larger training weights to underrepresented subgroups. These weights are used in a weighted binary cross-entropy loss.
- Each sample belongs to a joint subgroup defined by its target label y and sensitive attribute s.
- Subgroup weights are assigned inversely to subgroup size, using n_y,s as the subgroup sample count.
- The weighting scheme increases the contribution of underrepresented subgroups during training.
- The reweighted objective uses weighted binary cross-entropy, with each sample's subgroup weight multiplying its BCE term.
D. Gate Entropy Regularization
Gate entropy regularization promotes balanced expert utilization by encouraging higher-entropy routing distributions. It reduces concentrated routing while preserving diverse, interpretable expert contributions.
- The method adds gate entropy regularization to promote balanced utilization of experts.
- The regularizer encourages higher entropy in each input's routing distribution.
- By reducing overly concentrated routing, the regularizer encourages multiple experts to contribute during training.
- Preventing subgroup-specific expert monopolization keeps routing weights interpretable as diagnostics of subgroup allocation.
E. Training Objective
The training objective combines subgroup reweighting with gate entropy regularization in an end-to-end optimization of the encoder, experts, and gating network.
- The standard MoE, reweighted, and full variants selectively enable subgroup weights and entropy regularization.Standard MoE sets w_y,s = 1 and λ_ent = 0; the reweighted model uses subgroup weights with λ_ent = 0, while the full model uses both components.
- All variants are trained end-to-end with stochastic gradient descent while jointly optimizing the encoder, experts, and gating network.
IV. EXPERIMENTS
Experiments evaluate the approach on CelebA using binary attribute classification and fairness metrics, including Equalized Odds and FATS. The comparison spans multiple target and sensitive-attribute settings, with an ablation of the proposed components.
- Experimental Setup: The evaluation uses CelebA facial images with binary target and sensitive attributes.
- Comparative Evaluation: Table I compares fairness–accuracy trade-offs across multiple target attributes under male and young sensitive attributes, reporting EO, accuracy, and FATS.
- Experimental Setup: The MoE head contains four independently parameterized experts and a gating network producing a four-expert routing distribution.
- Experimental Setup: Performance is evaluated using classification accuracy, Equalized Odds, and the Fairness–Accuracy Trade-off Score.
- Ablation Study: Table II ablates the proposed components on CelebA for the attractiveness target.
B. Main Results
Across target and sensitive-attribute settings, the proposed method improves the fairness–accuracy trade-off over existing methods. Ablation results show that subgroup reweighting and gate entropy regularization jointly improve fairness and expert utilization.
- The proposed method consistently improves fairness while avoiding large accuracy drops compared with one-stage methods such as GRL, LNL, and MFD.
- It achieves a more favorable fairness–accuracy balance than two-stage methods, reflected in consistently lower FATS values.
- Ablation Study: 13.3 EO for standard MoE falls to 4.2 with subgroup reweighting, showing that data imbalance drives routing bias.
- Ablation Study: Figure 2 compares average gate weights across subgroups for standard MoE and the proposed method.
- Ablation Study: 4.1 EO after adding gate entropy regularization, which also recovers accuracy and produces balanced, interpretable routing.
D. Analysis of Expert Utilization
The standard MoE routes demographic subgroups unevenly, with some experts disproportionately activated and others underutilized. The proposed method mitigates this skew and provides a more balanced routing distribution.
- Standard MoE exhibits pronounced subgroup-dependent routing skew, with specific experts disproportionately activated by certain groups.
- The proposed method mitigates routing’s dependency on subgroup attributes and yields a more balanced distribution of routing weights.
- Under subgroup imbalance, conditional routing can skew expert utilization according to subgroup attributes and induce systematic bias.
- The routing distribution makes subgroup-level behavior interpretable by showing how experts are assigned across demographic groups.