PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Additive Experts: Context-Gated Experts for Controllable Model Additivity

Guangzhi Xiong, Sanchit Sinha, Aidong Zhang

arXiv:2602.10585v1cs.LGcs.AI

TL;DR

Interpretability and predictive accuracy remain difficult to reconcile because GAMs can miss complex feature relationships, while interactions can obscure feature-level attribution. NAEs address this gap with feature-specific experts, dynamic gating, and targeted regularization; theory and experiments report competitive accuracy with state-of-the-art black-box models while retaining transparent explanations.

  • Problem

    Strictly additive GAMs can limit predictive performance on complex relationships, whereas adding interactions can diminish transparent attribution to individual features.

  • Method

    NAEs associate each feature with specialized expert networks, use dynamic gating to integrate cross-feature information, and apply targeted regularization to control expert-prediction variance.

  • Results

    NAEs achieve competitive accuracy with state-of-the-art black-box models while offering transparent, feature-level explanations on synthetic and real-world datasets.

  • Takeaways & Limitations

    NAEs provide a principled mechanism for controlling the trade-off between predictive flexibility and feature-level transparency.

  • Takeaways & Limitations

    The theoretical analysis assumes dense expert routing, with all K experts available for each feature and gating over the full expert set.

Abstract

from arXiv · show

The trade-off between interpretability and accuracy remains a core challenge in machine learning. Standard Generalized Additive Models (GAMs) offer clear feature attributions but are often constrained by their strictly additive nature, which can limit predictive performance. Introducing feature interactions can boost accuracy yet may obscure individual feature contributions. To address these issues, we propose Neural Additive Experts (NAEs), a novel framework that seamlessly balances interpretability and accuracy. NAEs employ a mixture of experts framework, learning multiple specialized networks per feature, while a dynamic gating mechanism integrates information across features, thereby relaxing rigid additive constraints. Furthermore, we propose targeted regularization techniques to mitigate variance among expert predictions, facilitating a smooth transition from an exclusively additive model to one that captures intricate feature interactions while maintaining clarity in feature attributions. Our theoretical analysis and experiments on synthetic data illustrate the model's flexibility, and extensive evaluations on real-world datasets confirm that NAEs achieve an optimal balance between predictive accuracy and transparent, feature-level explanations. The code is available at https://github.com/Teddy-XiongGZ/NAE.

1 INTRODUCTION

NAEs address the interpretability–accuracy trade-off by extending additive models with specialized experts, dynamic gating, and targeted regularization. The framework is theoretically analyzed and empirically evaluated as a way to capture feature interactions while retaining transparent feature-level explanations.

  • Motivation and approach: NAEs are motivated by the limitation that strictly additive GAMs may miss complex feature relationships and that explicit interactions can make feature attribution less transparent.The introduction frames controllable additivity as a way to address this tension.
  • Motivation and approach: NAEs extend additive models with a mixture-of-experts architecture and dynamic gating to integrate feature interactions while preserving interpretability.Each feature is associated with specialized expert networks, while gating adaptively combines information across features.
  • Motivation and approach: Targeted regularization controls variance among expert predictions, providing a principled way to balance model flexibility and transparency.The regularization supports a transition from purely additive behavior toward models capturing nuanced feature interactions.
  • Analysis and evaluation: Theoretical analysis shows that NAEs can recover complex data-generating processes inaccessible to standard GAMs and clarifies regularization's role in the accuracy–interpretability trade-off.The analysis focuses on model flexibility and feature-level interpretability.
  • Analysis and evaluation: Experiments on synthetic and real-world datasets show competitive predictive performance with state-of-the-art black-box models while maintaining clear, feature-level explanations.The empirical validation covers both synthetic and real-world datasets.

2 NEURAL ADDITIVE EXPERTS

Neural Additive Experts associate each feature with multiple specialized predictors and use context-aware gating to combine them while preserving feature-wise contributions. Regularization and bounded attribution analyses provide control over flexibility, additivity, and interpretability.

  • Architecture: NAEs extend classical GAMs with multiple expert predictors per feature and a dynamic gating mechanism that captures context-dependent effects.The gating mechanism integrates information across features while the architecture remains organized around feature-specific outputs.
  • Dynamic Gating and Expert Aggregation: Gating scores determine expert relevance for each feature, and masking suppresses less significant experts before weighted aggregation.The masked relevance weights combine individual expert predictions into each feature’s output.
  • Prediction and Interpretability: The final prediction sums feature-specific aggregated outputs with an intercept, preserving feature-level decomposition despite context-dependent gating.This structure supports feature attribution while allowing interactions through the gating context.
  • Training Objective and Regularization: The expert variation penalty, weighted by λ, encourages consistency among experts and provides a tunable balance between modeling flexibility and interpretability.Task-specific loss functions and standard regularization are also used during training.
  • Interpretability and Feature Attribution: NAEs quantify each feature’s contribution and define upper and lower bounds that characterize the range and variability of its possible effects.Visualizing actual effects within these bounds provides insight into feature influence.
  • Feature Interactions: Pairwise interactions can be isolated from the gating function by removing other features’ influence and examining the resulting feature outputs.The resulting outputs reflect pairwise feature interactions captured by NAE.

3 THEORETICAL ANALYSIS OF NAES

The theoretical analysis characterizes NAE expressivity and its control over additivity. It establishes containment of additive and pairwise-interaction model classes while showing that increasing regularization drives NAEs toward additive behavior.

  • Assumptions: Under compact feature domains, continuous functions, per-feature encoders, expert functions, and softmax gating define the theoretical NAE setting.The analysis assumes dense routing in which all K experts are available for each feature.
  • GAM Containment: Any GAM can be represented exactly by an NAE with K = 1, so the NAE class contains generalized additive models.The construction uses unit routing and sets the composed expert function equal to each GAM component.
  • Separable Approximation: The expert construction represents a separable pairwise term using two experts with opposite outputs and a softmax gate dependent on the other feature.The gate can be restricted to depend on one specified feature by zeroing the remaining gating parameters.
  • GA2M Containment: For any GA2M function and ε > 0, an NAE exists whose uniform approximation error is less than ε.The construction uses separable approximations of pairwise terms and feature heads with multiple experts.
  • Expressivity: GAM ⊊ NAE(K) in uniform-norm closure when gates depend on other features, while NAE predictions remain feature-decomposed.This gives NAEs greater expressivity than vanilla GAMs and GA2Ms under the stated setting.
  • Monotone Additivity and Additive Limit: A(λ) is nondecreasing in λ and converges to 1 as λ approaches infinity, with limiting minimizers realized as GAMs.The penalty contracts within-feature expert outputs toward their mean, making gates irrelevant in the additive limit.

4 EXPERIMENTS ON SIMULATED DATA

Synthetic experiments test whether NAEs recover additive and multimodal shape functions and how λ controls flexibility versus additivity. NAEs recover multimodal structure that NAM misses, while increasing λ enforces additivity.

  • Simulation Setup and Visualization: NAEs and NAMs are evaluated on synthetic unimodal additive and multimodal non-additive datasets, with additional robustness studies in Appendix E.Each dataset contains 10,000 samples.
  • Simulation Setup and Visualization: Both models recover the underlying pattern in the unimodal setting, while NAE preserves interpretability without spurious complexity.
  • Simulation Setup and Visualization: In the multimodal setting, NAE recovers the interaction and multimodal structure, whereas NAM produces a near-linear fit.NAE’s upper and lower expert bounds reflect the range of feature contributions.
  • Effect of Variation Penalty λ: λ controls the flexibility–additivity trade-off: small values allow flexible recovery, while larger values converge toward strict additivity.Figure 3 examines λ values of 0.1, 1, and 10.
  • Effect of Variation Penalty λ: 0.597, 0.709, and 1.000 are the additivity scores for λ = 0.1, 1, and 10, respectively.

5 EXPERIMENTS ON REAL-WORLD DATA

Real-world evaluations compare NAEs with additive, interaction-based, and black-box baselines across six regression and classification datasets. NAEs maintain feature-level explanations while matching or exceeding the predictive performance of more complex models, with λ providing practical control over the accuracy–interpretability trade-off.

  • Datasets and Baselines: NAEs are evaluated on six datasets spanning regression and classification tasks, with varied sizes, feature types, and categorical variables.The datasets are Housing, MIMIC-II, MIMIC-III, Income, Credit, and Year.
  • Datasets and Baselines: The benchmark includes linear, spline, additive, interaction-based, and black-box models, including NAM, NBM, EBM, NODE-GAM, MLP, NODE, and XGBoost.
  • Predictive Performance and Feature Attribution: NAEs match or exceed the predictive performance of more complex models while maintaining clear, feature-level interpretability.Traditional additive models provide feature-level explanations but often underperform interaction-capable or black-box architectures.
  • Predictive Performance and Feature Attribution: NAE explanations require O(n) scalar feature contributions, whereas interaction-level GA2M explanations require O(n^2) components.Each NAE feature contribution is represented through a shape plot.
  • Predictive Performance and Feature Attribution: For Longitude effects, NAE shows context-dependent feature contributions and a Longitude–Latitude interaction associated with differing coastal and non-coastal effects.The plotted y-axis uses mean-centered feature contributions, with normalized data density shown in background bars and actual effects as blue dots.
  • Controlling the Trade-off Between Accuracy and Interpretability: On Housing, increasing λ improves additivity and tightness but may degrade RMSE, providing a practical mechanism to balance accuracy and interpretability.The authors use λ = 0.1 in their main experiments and suggest increasing it when greater interpretability is needed.

6 RELATED WORK

Related work improves GAM flexibility, additive explanations, or mixture-of-experts modeling, but existing approaches retain strict additivity or lack fine-grained control and feature-specific adaptation. NAEs address these limitations through architecture-level additive explanations, dynamic gating, and expert-specific regularization.

  • Generalized Additive Models: GAMs provide transparent feature-level attributions through sums of univariate functions, while newer variants improve flexibility, scalability, robustness, or uncertainty quantification.
  • Generalized Additive Models: GA2M captures interactions but can require many costly and difficult-to-visualize two-dimensional plots to interpret feature effects.
  • Generalized Additive Models: Additive SHAP explains black-box predictions post hoc, whereas NAE explanations are exact by construction and controlled during training through λ.NAE attributions are inherent to the forward pass rather than estimated afterward.
  • Mixture of Experts: Standard mixture-of-experts models use full-input experts and sacrifice the decomposability needed for interpretability, while NAEs organize experts per feature.
  • Mixture of Experts: Prior additive–mixture methods use fixed weights, shared weights, or lack explicit mechanisms to balance interpretability and accuracy.These limitations restrict adaptation or expressiveness across inputs and features.

7 CONCLUSION

The conclusion presents NAEs as a framework that combines mixture-of-experts flexibility and dynamic gating with feature-level interpretability. Theory and experiments support competitive accuracy, recoverable context-dependent relationships, and explicit control of the flexibility–transparency trade-off.

  • Conclusion: NAEs extend additive models with multiple specialized networks per feature and dynamic gating that relaxes strict additive constraints.
  • Conclusion: Targeted regularization provides explicit control over the trade-off between flexibility and transparency.
  • Conclusion: Theoretical analysis and experiments show that NAEs recover complex context-dependent relationships while achieving competitive accuracy with clear feature-level explanations.

Checklist

The paper’s checklist reports that theoretical assumptions, proofs, explanations, implementation details, empirical-result reporting, reproducibility materials, asset documentation, and human-subject disclosures are addressed as applicable.

  • The checklist marks mathematical setup, algorithmic complexity analysis, and anonymized source-code dependency specifications as included.
  • The checklist marks full assumptions, proofs, and explanations for theoretical results as included.
  • For empirical results, the checklist marks reproducible code and data, training details, measure definitions, error bars, and computing infrastructure as included.
  • Existing assets are documented with creator citations, license information, and applicable new-asset URLs.
  • Crowdsourcing and human-subject requirements are marked not applicable, including participant instructions, risks, compensation, and IRB links.

A DATASET DESCRIPTION

The evaluation uses six datasets spanning regression and classification, with varied sizes, feature types, and categorical-variable structure. Baselines, splits, optimization details, hyperparameter search, and selected NAE settings are specified.

  • Dataset statistics: Housing contains 20,640 instances with eight attributes, while Year contains 515,345 samples with 90 features.
  • Dataset statistics: MIMIC-II supports ICU mortality classification with 17 attributes, including seven categorical variables; MIMIC-III follows the NODE-GAM setting with dummy-coded categorical variables.
  • Dataset statistics: Credit is a fraud-detection task with 492 frauds among 284,807 transactions and 30 anonymized features, while Income predicts earnings above $50,000.
  • Evaluation setup: Baseline comparisons use officially released implementations, five-fold cross-validation for four datasets, and established train-validation-test splits with ten seeds for Housing and Year.
  • Implementation: NAE uses independently encoded features, MLP feature encoders, linear expert predictors, AdamW, cosine-annealed learning rates, and random hyperparameter search.The reported search includes dropout, output, variation, normalization, architecture, batch-size, iteration, learning-rate, and weight-decay choices.

D PROOFS FOR SECTION 3

The proofs establish that NAEs contain GAMs exactly, approximate GA2M functions with context-only gating, and become additive as the expert-variation penalty grows.

  • Approximation: Finite sums of separable continuous functions are dense in continuous pairwise functions by Stone–Weierstrass.
  • GAM representation: Any f in GAM can be represented exactly by an NAE using one expert per feature and constant gating.
  • GA2M approximation: Any continuous GA2M function can be approximated arbitrarily well by an NAE with context-only gating and finitely many experts.The construction realizes univariate terms directly and separable pairwise products through gated experts.
  • Regularization: The expert-variation penalty is nonincreasing with λ and converges to zero as λ tends to infinity under the stated nonnegative-loss setting.
  • Additivity limit: As λ increases without bound, expert outputs collapse at training points and the additivity metric A approaches 1.At zero penalty, identical expert outputs admit a GAM realization with the same predictions.

E.1 Impact of Feature Sparsity

Synthetic studies show that NAEs handle sparse, multimodal, correlated, and non-additive data while retaining controllable explanations. Variants and ablations clarify how gating, expert counts, and regularization affect the flexibility–additivity trade-off.

  • E.1 Impact of Feature Sparsity: NAEs recover minority patterns under feature sparsity, whereas traditional GAMs overfit the majority class; more experts consistently recover multimodal structure.The sparsity study varies the minority proportion across 50%, 25%, 5%, and 1%, with K = 2, 8, and 32 experts.
  • E.2 Distributional Complexity: NAEs capture increasingly complex shape functions as the number of categorical features and modes grows.The simulation varies the number of categorical features across CF = 1, 3, 5, and 7.
  • E.3 Impact of Correlated Features: NAE maintains low RMSE across correlation levels, while NAM performs significantly worse in the correlated-feature simulation.
  • E.4 Impact of Generic Interactions: 0.1306 RMSE for NAE versus 1.0114 for NAM on a multiplicative interaction that cannot be decomposed additively.
  • Interpretability: NAE provides feature attributions and interaction visualizations, while EB2M’s single-feature attribution bounds can be too loose to interpret.
  • Higher-order interactions: NAE can represent three-way interactions, with spatially varying AveRooms effects beyond Latitude–Longitude interactions.
  • NAE-D: NAE-D approaches traditional additive-model performance when scoring matrices are regularized to be diagonal and provides prediction bounds beyond point estimates.
  • NAE-E: NAE-E’s performance is slightly worse than NAE because of its discrete range, although its estimated bounds remain tight without variation penalty.

J DISCUSSION ON MODEL COMPLEXITY

NAE extends NAM with expert encoding and dynamic routing, increasing computational and memory costs to represent feature interactions. Theoretical and empirical analyses characterize these costs, while experiments evaluate scaling and explanation visualizations across real-world datasets.

  • Model complexity: NAE introduces additional parameters in expert encoding and dynamic routing compared with neural additive models.The complexity analysis is framed relative to Neural Additive Models (NAMs), using n input features.
  • Theoretical and empirical costs: NAE’s routing cost includes a squared term in the number of features, unlike NAE-D’s linear additional cost.NAE uses a full n × n block matrix for feature interactions, whereas NAE-D simplifies this to a block-diagonal matrix.
  • Empirical scaling: NAE training time and memory usage increase with both the number of features and experts.These measurements use 5000 training samples, 100 epochs, and a four-layer encoder with 128 neurons per layer.
  • Baseline comparison: With K = 4, NAE preserves interaction modeling while incurring lower computational cost than the compared end-to-end baselines.The comparison includes NA2M, EB2M, NAM, and EBM; experiments exceeding one hour are marked with “–”.
  • Explanation visualizations: Figures 11–14 visualize NAE feature explanations for the Housing, MIMIC-II, Income, and Credit datasets.The displayed real-world datasets each contain no more than 50 features.
Loading 2602.10585v1…