PaperlayerView exact revision on arXiv

Source-linked AI summary

NODE-GAM: Neural Generalized Additive Model for Interpretable Deep Learning

Chun-Hao Chang, Rich Caruana, Anna Goldenberg

arXiv:2106.01613v3cs.LG

TL;DR

High-stakes machine-learning settings require interpretability alongside accuracy, while GAMs lack deep learning’s differentiability and scalability. NODE-GAM and NODE-GA2M combine neural models with GAM forms; they perform well against other GAMs, including on larger datasets and limited-label settings.

  • Problem

    GAMs are interpretable and used in high-risk domains, but lack deep learning’s differentiability and scalability, while such settings also require fairness, robustness, and inspectable predictions.

    Source: §1 · p. 1

  • Method

    The paper modifies NODE to create interpretable neural GAM and GA2M models, using gating to reduce higher-order interactions and select main and pairwise features.

    Source: §1 · p. 2

  • Results

    NODE-GAM and NODE-GA2M perform similarly to other GAMs on medium datasets and outperform them on larger datasets; self-supervised pretraining improves performance when labeled data are limited.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    The approach targets high-risk and large-data settings where GAMs have been useful but faced scalability or accuracy issues, and also addresses settings with limited labeled data.

    Source: §1 · p. 2

    Patterns shown in the shape graphs are associations, not causal effects, and require a proper causal study for validation.

    Source: §5 · p. 9

Abstract

from arXiv · show

Deployment of machine learning models in real high-risk settings (e.g. healthcare) often depends not only on the model's accuracy but also on its fairness, robustness, and interpretability. Generalized Additive Models (GAMs) are a class of interpretable models with a long history of use in these high-risk domains, but they lack desirable features of deep learning such as differentiability and scalability. In this work, we propose a neural GAM (NODE-GAM) and neural GA$^2$M (NODE-GA$^2$M) that scale well and perform better than other GAMs on large datasets, while remaining interpretable compared to other ensemble and deep learning models. We demonstrate that our models find interesting patterns in the data. Lastly, we show that we improve model accuracy via self-supervised pre-training, an improvement that is not possible for non-differentiable GAMs.

1 INTRODUCTION

GAMs offer interpretable models for tabular data, but traditional approaches lack deep learning’s differentiability and scalability. NODE-GAM and NODE-GA2M adapt NODE to retain interpretability while improving scalability and performance, including when labeled data is limited.

1 INTRODUCTION

  • High-stakes uses call for fairness, bias discovery, robustness under distribution shifts, and models whose predictions can be inspected and understood.The authors identify healthcare as a setting where GAMs have been useful but faced scalability or accuracy issues on newer large datasets.Source: §1 · pp. 1–2
  • GAMs have a long history in fields including medicine, business, and ecology, and tree-based GAMs and GA2Ms improve accuracy and pattern discovery over spline GAMs.GAMs are also used to detect dataset bias and audit black-box models.Source: §1 · p. 1
  • NODE-GAM and NODE-GA2M are comparable to deep-learning approaches on tabular data, outperform other GAMs on larger datasets, and benefit from self-supervised pretraining when labels are limited.The contributions summary also reports similar performance to other GAMs on medium-sized datasets.Source: §1 · pp. 1–2
  • Other interpretable tabular approaches have trade-offs: attention-based feature importance may not align with input importance, and NIT uses longer two-stage iterative training.NIT’s performance is described as slightly below DNNs, while the proposed models are overall on par with them.Source: §1 · p. 1
  • NODE is differentiable and performs strongly on tabular data but lacks interpretability, while NAM is interpretable but lacks pairwise interactions and may require substantial computation.NAM may train and ensemble tens to hundreds of models, whereas the proposed model trains once.Source: §1 · p. 2
  • The paper adapts NODE into neural GAM and GA2M models, aiming to combine the benefits of deep learning with interpretable model forms.GPU optimization and mini-batch training support scaling to larger datasets.Source: §1 · pp. 1–2
  • Gating mechanisms reduce higher-order feature interactions and let back-propagation select main effects and pairwise features automatically.This avoids the additional pair-selection algorithm used by tree-based GA2M.Source: §1 · p. 2

2 BACKGROUND

GAMs and GA2Ms represent predictions through feature effects and pairwise interactions that can be visualized. NODE builds on differentiable oblivious decision trees, whose outputs can be stacked across layers.

2 BACKGROUND

  • GAMs and GA2Ms express predictions using feature main effects and, for GA2Ms, pairwise interactions; their graphs show each feature’s contribution for a given input value.A feature graph plots x_j against f_j(x_j), and the effects can be read and added to simulate model behavior.Source: §2 · p. 2
  • The GAM baselines are Explainable Boosting Machine for tree-based GAM and GA2M models and cubic splines implemented in pygam.Source: §2 · p. 2
  • NODE consists of L layers, each containing I differentiable oblivious decision trees of equal depth C.An oblivious decision tree shares its input features and thresholds among nodes at the same depth, allowing parallel computation.Source: §2 · pp. 2–3
  • NODE replaces nondifferentiable feature selection and threshold indicators with entmax and entmoid transformations so the tree computations are differentiable.Entmax provides sparse feature weights, while entmoid produces values between 0 and 1.Source: §2 · p. 3
  • NODE passes each layer’s tree outputs into the next layer alongside the input features, then averages tree outputs across layers for its final prediction.The layer input is defined as the original input combined with outputs from preceding layers.Source: §2 · p. 3

3 OUR MODEL DESIGN

NODE-GAM constrains NODE to additive, feature-wise trees, while NODE-GA2M extends the design to pairwise interactions. The section also describes attention, output aggregation, regularization, and shape-graph extraction.

  • NODE-GAM makes each tree select one feature and restricts cross-layer connections to trees focused on that same feature.Trees at each layer produce outputs that are concatenated into the final linear layer.Source: §3 · p. 3, Figure 1 · p. 4
  • Annealing temperature T from 1 to 0 over S steps makes entmax gates gradually one-hot, so connection gates retain only matching feature selections.Source: §3 · p. 3
  • Attention-based GAM adds weights over prior trees, constrained to eligible same-feature connections; factorizing logits as A=BC reduces memory using embedding dimension E.Source: §3 · p. 4
  • The final linear layer learns a weighted sum of tree outputs; regularization adds dropout, feature subsampling, and L2 penalties, with a class-prior offset for imbalanced binary labels.Source: §3 · p. 4
  • NODE-GA2M uses two alternating feature logits per tree, permitting at most two interacting features and connecting trees only when their feature pairs match.Source: §3 · p. 5
  • Shape graphs isolate GAM main effects by comparing examples differing only in one feature, while GA2M graphs aggregate trees by effect and purify interactions into main effects when possible.Source: §3 · p. 5

4 RESULTS

The results compare NODE-GAM and NODE-GA2M with other GAMs and full-complexity models, examine the patterns their shape graphs reveal, and test self-supervised pre-training when labels are limited. On large datasets, NODE-GAM slightly exceeds EBM on datasets where EBM runs, while NODE-GA2M outperforms EBM-GA2M by up to 7.3% in Higgs.

4.1 ARE NODE-GAM AND NODE-GA2M ACCURATE?

  • On eight medium-sized datasets, GAMs perform similarly; Spline leads in Rank, NODE-GAM in NS, and NODE-GA2M is similar to EBM-GA2M and slightly exceeds full-complexity models.For full-complexity models, XGB performs best and RF worst; GAM differences are often smaller than the standard deviation.Source: §4.1 · pp. 5–6, Table 1 · p. 6
  • On six datasets with over 500K samples, NODE-GAM runs slightly better than EBM on five datasets, while NODE-GA2M outperforms EBM-GA2M by up to 7.3% in Higgs and 3.6% on average.Spline runs out of memory on five of six datasets, and EBM cannot run on Epsilon with 2K features.Source: §4.1 · p. 6, Table 2 · p. 6

4.2 SHAPE GRAPHS OF NODE-GAM AND NODE-GA2M: BIKESHARE AND MIMIC2

  • In Bikeshare shape plots, all four GAMs agree on commute-hour peaks and fewer rentals at high temperatures, while NODE-GA2M differs on colder-weather and summer patterns.Its four illustrated interactions include commute patterns by working day and weekday, temperature-dependent morning and afternoon rentals, and lower rentals at high humidity.Source: §4.2 · p. 7, Figure 2 · p. 7, Figure 3 · p. 7
  • On MIMIC2, model shape plots show rising risk with age and bilirubin, higher risk at GCS=1 than GCS=5, and PFratio drops at imputed mean 332 and the healthy-range threshold above 400.The sharp PFratio drop at 332 reflects missing values imputed with the population mean; Spline does not capture it.Source: §4.2 · p. 8, Figure 4 · p. 8
  • MIMIC2 interactions adjust risk beyond main effects: high bilirubin raises risk for younger patients but lowers the interaction effect for patients over 80, while GCS interactions alter risk by PFratio and age.For PFratio above 400, the interaction raises risk for GCS=1,2 and lowers it for GCS=3,4,5.Source: §4.2 · p. 8, Figure 5 · p. 8

4.3 SELF-SUPERVISED PRE-TRAINING

  • With limited labeled data, self-supervised NODE-GAM improves over No-SS on six of seven datasets, with the largest improvement of 6% on MIMIC2, and beats EBM on three of six datasets by up to 10% on Churn.The comparison covers labeled-data ratios from 0.5% to 10%; Credit at 0.5% is excluded because it has too few positive samples.Source: §4.3 · p. 9, Figure 6 · p. 9

5 LIMITATIONS, DISCUSSIONS AND CONCLUSIONS

NODE-GAM and NODE-GA2M extend GAMs to deep learning while retaining the interpretability assumed for the GAM class. The authors caution that patterns in their shape graphs are associations, not causal findings.

5 LIMITATIONS, DISCUSSIONS AND CONCLUSIONS

  • Patterns shown in the shape graphs should be treated as associations, not causal relationships; claims based on them require proper causal validation.Source: §5 · p. 9
  • The paper assumes GAMs are interpretable and therefore treats NODE-GAM as interpretable to the same degree as other GAMs.It cites user studies reporting that GAMs can help doctors interpret models and help users understand patterns and feature importance.Source: §5 · p. 9
  • NODE-GAM and NODE-GA2M automatically learn which main effects and pairwise interactions to focus on, without two-stage training or model ensembles.The authors report greater accuracy than traditional GAMs on large datasets and in limited-labeled settings.Source: §5 · p. 9

ETHICS STATEMENT

The ethics statement warns against interpreting GAM patterns as causal and describes their use in inspecting potential dataset biases. The reproducibility statement provides code and details for reproducing the results.

ETHICS STATEMENT

  • GAM visualizations can help users confirm known biases and examine hidden biases, but learned patterns can reflect dataset bias rather than causal relationships.The authors warn that a misleading interpretation could claim asthma benefits pneumonia patients.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The authors released code with instructions and hyperparameters, plus included code to download datasets and reproduce train/test folds.Dataset details and best-performing model and baseline hyperparameters are documented in supplementary sections F–H.Source: REPRODUCIBILITY STATEMENT · p. 10

B THE PERFORMANCE OF NODE-GA2M AND TABNET WITH DEFAULT

The supplement reports NODE-GA2M default-hyperparameter performance, provides model pseudocode, describes GA2M purification, and illustrates the NODE-GA2M architecture.

B THE PERFORMANCE OF NODE-GA2M AND TABNET WITH DEFAULT

C PSEUDO-CODE FOR NODE-GAM

D PURIFICATION OF GA2M

  • Purification addresses the ambiguity of equivalent GA2M representations by moving interaction effects into main effects when possible.The described procedure bins continuous features and moves average interaction effects into their corresponding main effects.Source: D PURIFICATION OF GA2M · p. 13

E NODE-GA2M FIGURES

  • The architecture illustration shows layers of differentiable oblivious trees, each depending on at most two features, with connections limited to trees sharing the same feature pair.Outputs from all layers are concatenated as inputs to the final linear layer.Source: E NODE-GA2M FIGURES · p. 15

F DATASET DESCRIPTIONS

The described datasets span prediction tasks in healthcare, business, finance, transportation, and wine quality, alongside large benchmark datasets used in prior NODE work.

F DATASET DESCRIPTIONS

  • Churn predicts telecom subscription attrition; Support2 and MIMIC II/III concern hospital mortality, with MIMIC covering ICU patients at a Boston academic medical center.Income uses 1994 census data to predict income above $50K annually; Credit predicts fraud from privacy-protecting PCA features.Source: F DATASET DESCRIPTIONS · p. 15
  • Bikeshare records hourly rental counts in Washington, D.C., while Wine predicts wine quality from laboratory measurements.Source: F DATASET DESCRIPTIONS · pp. 15–16
  • The authors use NODE-paper scripts for six datasets and provide included code to download datasets into train and test folds.The section says it describes eight datasets and summarizes them in a table.Source: F DATASET DESCRIPTIONS · pp. 15–16
  • The listed large benchmark datasets include Higgs, Yahoo, and Year, with Higgs and Year identified as UCI datasets and Yahoo linked to Yahoo Webscope.Source: F DATASET DESCRIPTIONS · p. 16

F.1 PREPROCESSING

Categorical features are target-encoded, and all features are quantile-transformed to a Gaussian distribution. Small Gaussian noise during fitting is reported as crucial for the transformed data's mean and standard deviation.

F.1 PREPROCESSING

  • All features are quantile-transformed with 2,000 bins to a Gaussian distribution, which the authors found performed better than a uniform distribution.Small Gaussian noise (e.g., 1e-5) is added during fitting, but not during transformation, to obtain mean 0 and standard deviation close to 1.Source: §F.1 · p. 16
  • Categorical features are target-encoded, following Popov et al. (2019).The procedure is used for NODE and NODE-GAM/GA2M.Source: §F.1 · p. 16
  • The appendix identifies sklearn.preprocessing.quantile_transform as the quantile-transform implementation.Table 6 is captioned as containing all dataset statistics and descriptions, but its values are not included here.Source: §F.1 · p. 16, Table 6 · p. 17

G HYPERPARAMETERS SELECTION

Hyperparameters are tuned using validation data, with different split and reporting procedures for medium-sized datasets and datasets from the NODE paper. The appendix also records the settings used for EBM, spline, neural, XGBoost, and random-forest methods.

G HYPERPARAMETERS SELECTION

  • Medium-sized datasets use five-fold cross-validation across five test splits; NODE-paper datasets use the authors' train/validation/test split and report seed-based standard deviations.Hyperparameters are tuned only on the first medium-dataset fold to reduce computational overhead; NODE models use one TITAN Xp GPU, four CPUs, and 8GB memory, while EBM and Spline use 32 CPUs and 120GB memory.Source: G HYPERPARAMETERS SELECTION · p. 17

G.1 EBM

  • EBM uses 100 inner and outer bags and up to 20,000 rounds; EBM GA2M selects among 16, 32, 64, or 128 interactions on validation data.For large datasets, EBM GA2M uses 64 interactions.Source: §G.1 · p. 17

G.2 SPLINE

  • Spline uses 50 cubic-spline knots per feature and selects the best lambda penalty from 1e-3 to 1e3 over 15 trials.The authors report that using more than 50 knots would crash the model.Source: §G.2 · p. 17

G.3 NODE, NODE-GA2M AND NODE

  • NODE-family training uses QHAdam, averages the most recent 5 checkpoints, warms learning rate for 500 steps, early-stops after 11k unimproved steps, and decays it to 1/5 after 5k unimproved steps.Batch size is 2,048 or the largest batch fitting GPU memory, with a minimum of 128.Source: §G.3 · pp. 17–18

G.4 XGBOOST

  • For XGBoost on medium-sized datasets, depth is 3, learning rate is 0.1, and training uses 50,000 estimators with 50-round early stopping.For large datasets, the paper reports the original NODE-paper performance.Source: §G.4 · p. 18

G.5 RANDOM FOREST (RF)

  • Random Forest uses sklearn's default hyperparameters with 1,000 trees.Source: §G.5 · p. 18

H BEST HYPERPARAMETERS FOUND IN EACH DATASET

The appendix reports the best hyperparameters found for NODE-GAM on nine medium-sized datasets. Table 7 is identified as the NODE-GAM hyperparameter table.

H BEST HYPERPARAMETERS FOUND IN EACH DATASET

I COMPLETE SHAPE GRAPHS IN BIKESHARE AND MIMIC2

The appendix presents shape plots and feature importance for main effects and selected interactions in Bikeshare and MIMIC2. Each dataset has plots for all main effects and its top 16 interactions.

I COMPLETE SHAPE GRAPHS IN BIKESHARE AND MIMIC2

Related papers

Loading 2106.01613v3…