PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Auditable Classifier Models: Source-Disjoint Tree Ensembles

Srikumar Krishnamoorthy

arXiv:2608.15725v1cs.LG

TL;DR

Clinical classifiers need both accuracy and auditability, but existing ensembles and interpretable alternatives make that balance difficult. RPTE separates feature discovery, source-disjoint tree construction, and coefficient estimation, achieving competitive performance while substantially reducing audit complexity.

  • Problem

    Clinical and regulated applications need accurate classifiers whose predictions remain transparent and verifiable for audit.

  • Method

    RPTE separates symbolic feature discovery, source-disjoint shallow-tree construction, and sparse coefficient estimation into three stages.

  • Results

    Mean ROC AUC was 0.830 across twelve benchmarks, within 0.8 percentage points of the best opaque baseline, while model inspection units fell 9×–87× versus XGBoost.

  • Takeaways & Limitations

    RPTE provides competitive predictive performance with substantially lower structural audit complexity across the evaluated clinical benchmarks.

  • Takeaways & Limitations

    The audit-load metrics measure structural workload rather than auditors’ cognitive effort, so domain-expert user studies are needed.

Abstract

from arXiv · show

Predictive models in clinical and regulated settings must be accurate and fully auditable. Tree ensembles deliver strong accuracy on tabular data, but their sequential boosting couples structure discovery with coefficient estimation, making compact per-prediction auditing difficult. Interpretable alternatives impose structural constraints that limit expressiveness: generalized additive models typically restrict interactions to pairwise terms and post-hoc rule extractors produce overlapping rules that hinder compact interpretation. We introduce Residual Pattern Tree Ensemble (RPTE), a three-stage learning approach, that is built on three key principles: bounded feature budget, source disjointness, and separate coefficient estimation. Stage~1 builds a supervised symbolic feature vocabulary. Stage~2 grows shallow trees under a source-disjointness constraint, where each raw variable is allocated to at most one tree, and retains only the discovered tree structures. Stage~3 solves a single $\ell_1$-regularized logistic regression over leaf-region indicators, yielding jointly optimal sparse coefficients. This learning approach ensures that every prediction decomposes into an algebraic sum of named, non-overlapping rule contributions, enabling full auditability by design. Empirical evaluation on twelve clinical-domain binary classification benchmarks using repeated stratified 5-fold cross-validation shows that RPTE performs competitively against tuned opaque ensembles and interpretable baselines. RPTE reduces model inspection units by 9$\times$ to 87$\times$ relative to XGBoost and maintains lower audit complexity than EBM on all 12 datasets. RuleFit requires comparable or fewer inspection units on three datasets where its rule count is small, but without source-disjointness guarantees. The source code is available at \href{https://github.com/srikumar2050/hugiml-core}{this https URL}.

1 Introduction

RPTE addresses the auditability challenges of accurate tree ensembles by separating structure discovery from coefficient estimation and enforcing bounded, source-disjoint representations. The study formalizes these principles, establishes complexity bounds, and evaluates predictive and interpretability performance across twelve clinical-domain binary classification datasets.

  • Motivation: Regulated-domain models must combine strong classification accuracy with transparent, traceable decision logic for regulatory review.This tension motivates auditable predictive modeling in healthcare, finance, and insurance.
  • Problem: Sequential boosted trees achieve strong tabular accuracy but make individual predictions difficult to trace to a single readable set of rules.Variables may recur across many trees and thresholds, requiring reviewers to reconstruct multiple decision paths.
  • Related work: Existing interpretable alternatives trade off expressiveness or compactness: EBMs automatically detect only pairwise interactions, while RuleFit can miss interactions and produce conflicting rule contributions.Ordered rule lists avoid simultaneous activation but depend on list length and preceding-rule failures.
  • Approach: RPTE uses a bounded symbolic feature vocabulary, source-disjoint tree structures, and separate structure discovery from coefficient estimation.Its three-stage approach first mines patterns, then retains shallow source-disjoint tree structures, and finally estimates coefficients jointly with sparse regularized regression.
  • Contributions: RPTE is evaluated on twelve clinical-domain binary classification datasets against ensemble methods and interpretable baselines, including interpretability, Rashomon, ablation, and robustness analyses.The contributions also include five structural properties and corresponding complexity bounds that provide a formal foundation for auditability.

2 Related Work

Prior work spans boosted and bagged tree ensembles, additive interpretable models, rule-based classifiers, ordered rule lists, and sparse structural-basis methods. These approaches motivate RPTE’s combination of source-disjoint structure discovery, higher-order interaction search, and joint leaf-value estimation.

  • Gradient-boosted ensembles: Boosted and bagged tree ensembles couple structure discovery with local leaf-value estimation, whereas RPTE discards discovered leaf predictions and jointly re-estimates leaf values.Gradient boosting fits trees sequentially to pseudo-residuals, while Random Forests aggregate independently randomized deep trees.
  • Additive interpretable models: Generalized additive models and EBMs are transparent by construction but restrict automatic interactions to individual variables or pairs, with higher-order terms requiring explicit specification.Neural additive models replace shape functions with small networks while retaining the additive-model framework.
  • Rule-based interpretable classifiers: RuleFit extracts conjunctive Random Forest rules and applies lasso regression, but lacks source disjointness and inherits CART’s marginal-screening limitation.The proposed Stage 2 instead uses pairwise-composition search to discover higher-order interactions at polynomial cost.
  • Other interpretable classifiers: Other interpretable families include ordered rule lists, sparse scoring systems, maximum-likelihood rule ensembles, sequential covering, optimal classification trees, and Rashomon-set methods.These methods represent additional precedents for compact or structurally constrained classification models.
  • Structural-basis methods and gaps: Prior structural-basis methods estimate sparse models over discovered components, but RPTE builds its basis through residual-guided source-disjoint search and fits structural indicators after discarding leaf values.The review identifies gaps in combining source disjointness with separated estimation and automatic higher-order interaction discovery without a fixed low-order cap.

3 Preliminaries and Notations

This section defines the raw data, labels, feature vocabulary, source ownership, and construction limits used by RPTE. It also formalizes audit-load metrics, with MIU as the primary model-level measure and IIU as per-prediction audit cost.

  • Notation: The raw input is X ∈ R^n×p0 with n observations and p0 variables, paired with binary labels y ∈ {0, 1}^n.
  • Feature vocabulary: The feature vocabulary combines raw features, up to K mined conjunctive patterns, and optional up to K pairwise arithmetic transforms.Mined patterns involve at most two raw variables, while the user-specified budget K independently bounds pattern and augmented-pair counts.
  • Source ownership: Each vocabulary column owns at most two raw variables, and two columns are source-disjoint when their source sets do not intersect.Raw features own themselves; derived patterns and pairs own the raw variables from which they are constructed.
  • Model construction: Vocabulary construction occurs inside each cross-validation training fold to prevent label leakage, while accepted trees satisfy M ≤ min(T, p0).The bound follows because each accepted tree claims at least one raw variable; T is the maximum number of boosting rounds.
  • Audit load: MIU is the primary model-level audit metric, whereas IIU measures mean conditions consulted per test prediction; both quantify auditor workload rather than condition difficulty.MU counts non-zero final coefficients, MIU counts model-wide split and direct-term conditions, and IIU captures per-prediction audit cost.

4 Methodology

RPTE separates feature-vocabulary construction, source-disjoint tree-structure discovery, and global coefficient estimation into three stages. Its bounded budget, non-overlapping raw-variable ownership, and jointly optimized sparse coefficients make predictions auditable by design.

  • Three-stage learning: RPTE explicitly separates primitive selection, tree composition, and predictive-weight estimation across three learning stages.Stage 1 builds a supervised vocabulary; Stage 2 discovers tree structures; Stage 3 estimates coefficients globally.
  • Bounded feature budget: V ≤ p0 + 2K bounds the supervised vocabulary, while tree limits make maximum audit load computable before training.The structural bounds include T rounds, Lmax leaves, and Dmax depth, making audit cost deterministic rather than emergent.
  • Source disjointness: Each raw variable is allocated to at most one tree, yielding prediction terms with non-overlapping raw-variable sources.After a tree uses a source, subsequent trees cannot use columns derived from that source; reviewers can audit predictions by summing active leaf-path coefficients.
  • Separated coefficient estimation: Stage 2 discards temporary Newton leaf values after discovering accepted structures, and Stage 3 jointly estimates sparse coefficients with ℓ1-regularized logistic regression.Leaf indicators and unused vocabulary columns form the design matrix, while the ℓ1 penalty drives some coefficients to zero.
  • Structure discovery: Accepted trees are selected by backtracking for binomial-deviance reduction, then their used raw sources are removed from the pool for later rounds.Tree construction terminates when no eligible split or deviance-reducing step is found.

5 Theoretical Analysis

Theoretical results show that RPTE training descends monotonically, terminates after finitely many source-disjoint rounds, and yields bounded, auditable prediction complexity. Its Stage 3 optimization has a unique global solution under full column rank.

  • Training guarantees: Training deviance strictly decreases at every accepted Stage 2 round, and the algorithm accepts at most min(T, p0) rounds.A round is accepted only when backtracking finds η_m > 0 that lowers deviance; otherwise Stage 2 terminates.
  • Structural guarantees: Accepted trees are pairwise source-disjoint because each round removes the accepted tree’s raw-variable sources from the available set.Rejected trees do not modify the available-source set.
  • Prediction decomposition: Each prediction contains at most M + |D| active contributions, with one leaf contribution per accepted tree and at most one contribution per retained direct term.The default construction separates tree sources from direct-term sources; strict direct-term ownership makes every active contribution fully source-disjoint.
  • Complexity bounds: O(V · n) is the per-round tree-growth cost for fixed Lmax and Dmax, where V is vocabulary size and n is the number of observations.The unrestricted-depth expression is O(Dmax V n), which simplifies to O(V n) when Dmax is fixed.
  • Model complexity: MIU ≤ M · Lmax · Dmax + |D|, while ℓ1 sparsification admits an optimum with at most rank(Z) ≤ min(n, q) active terms, where q = M · Lmax + |D|.The leaf-region design matrix has at most M · Lmax indicators, and M ≤ min(T, p0).
  • Stage 3 optimality: With full column rank in Z, Stage 3 is strictly convex and coordinate descent converges to the unique global minimum.Without full column rank, coefficient minimizers may differ, but they produce identical predicted probabilities.

6 Experimental Evaluation

Across twelve clinical-domain binary classification benchmarks, RPTE matched several baselines in predictive performance while substantially reducing audit complexity. Its strictness and component ablations quantify the tradeoffs among accuracy, compactness, and source-disjoint construction.

  • Experimental setup: RPTE was evaluated on twelve datasets using repeated stratified 5-fold cross-validation with 15 outer splits, compared against XGBoost, LightGBM, RF, EBM, and RuleFit.All six models used uniform grids of eight hyperparameter configurations; metrics included ROC AUC, balanced accuracy, Brier score, and MIU/IIU.
  • Predictive performance: 0.830 mean AUC placed RPTE below XGBoost at 0.838 and EBM at 0.853, but differences versus LightGBM, RF, and RuleFit were not significant.RPTE achieved perfect AUC on corral and 0.990 ROC AUC on EEG-Seizure, only 0.006 below LightGBM; Holm-corrected p-values were 0.442, 0.467, and 0.700, respectively.
  • Auditability: 9× to 87× XGBoost-to-RPTE MIU ratios showed lower RPTE audit load on all twelve datasets, while RPTE also remained below EBM on all twelve.EBM reached 145,155 MIU on EEG-Seizure, whereas RPTE-to-EBM ratios ranged from 2× on postop. to 328× on EEG-Seizure.
  • Accuracy–auditability tradeoff: 72 mean MIU (median 37) made RPTE the lowest-audit-load model, versus 78 (68) for RuleFit; opaque ensembles improved mean AUC by only 0.5–0.8 percentage points while requiring 25×–121× greater mean MIU.EBM’s complete model-level MIU was approximately 189× higher by the mean and 14× higher by the median, despite higher AUC and comparable prediction-level IIU.
  • Constraint ablation: 1.8 percentage points lower mean AUC accompanied the strict constraint, which reduced active direct terms from 2,171 to 243 across 180 folds, an 89% reduction.The AUC, balanced accuracy, and Brier-score reductions were statistically significant after Holm correction, making the stricter constraint suited to complete variable ownership or reduced audit complexity.
  • Component ablations: 2.1× lower mean MIU resulted from HUG versus the frequent vocabulary, while Stage 3 global refitting improved mean AUC by 2.2 percentage points over retaining Stage 2 leaf values.Full RPTE beat the original-only variant on 11 of 12 datasets, and leaf reassignment plus refitting won on 10, tied on one, and lost on one.

7 Discussion, Implications, and Limitations

RPTE trades a modest amount of predictive performance for substantially lower, source-disjoint audit complexity, while discovering clinically aligned and sometimes alternative near-equivalent explanations. Its deployment value is strongest when compact human review matters, but conclusions are limited by the dataset-level comparison and bounded Rashomon analysis.

  • Predictive performance: RPTE averages 0.830 AUC, below XGBoost (0.838) and EBM (0.853), while differing nonsignificantly from LightGBM, RF, and RuleFit.Holm-corrected comparisons are significant against XGBoost and EBM (pHolm = .005 and .039), but not against LightGBM, RF, or RuleFit.
  • Audit complexity: 9× to 87×: RPTE’s XGBoost-to-RPTE MIU ratio spans this range, and RPTE has lower MIU than EBM on all 12 datasets.EBM reaches 12,593 MIU on WDBC and 145,155 on EEG-Seizure, reflecting feature-wise shape functions and additional interaction terms.
  • Audit complexity: RuleFit has comparable mean MIU (78 vs. 72) and lower MIU on three datasets, but lacks source disjointness and can produce overlapping, conflicting rule effects.A RuleFit model with MIU = 64 may activate 20 rules per prediction with shared variables and conflicting coefficient signs.
  • Discovered structure: RPTE repeatedly discovers clinically aligned structures: cardiac pathways in all 15 heart-c folds, two breast-cancer morphology dimensions in all 15 WDBC folds, and a dominant SA Heart signature in 10 of 15 folds.The SA Heart architecture shows active-tree stability of 0.72 and prediction Spearman of 0.82.
  • Alternate explanations: RPTE can expose multiple explanations for nearly the same patient groups, including coverage Jaccard values of 1.000 on WDBC and heart-c under same-fold feature substitutions.Fixed-partition comparisons show these alternatives arise from fitting different feature sets rather than from changes in sampled patients.
  • Implications and limitations: Approximately 75 MIU on heart-c makes comprehensive human review more tractable, while Stage 1 vocabulary construction supplies the largest accuracy gain and HUG yields more compact models at comparable AUC.Source disjointness imposes a modest accuracy cost for a substantial audit-load reduction; statistical comparisons use 12 dataset-level observations, and the Rashomon analysis covers a bounded set.

8 Conclusion and Future Research

RPTE separates structure discovery from coefficient estimation, retaining competitive predictive performance while reducing audit complexity. Future work includes alternative representations and penalties, scalable Rashomon-set search, and broader validation.

  • Conclusion: RPTE separates structure discovery from coefficient estimation and achieves a mean ROC AUC of 0.830 across twelve benchmarks.Its performance is within 0.8 percentage points of the best opaque baseline.
  • Conclusion: Gradient-boosted ensembles remain preferable when maximum accuracy is the sole objective, while RPTE targets higher-order interactions requiring full per-prediction auditability.EBMs remain an alternative when pairwise interactions suffice and per-feature shape functions are preferred.
  • Future research: Future research will examine alternative Stage 1 vocabularies, structured Stage 3 penalties, scalable Rashomon-set search, and external, multi-site, and temporal validation.Candidate penalties include elastic net and group lasso.

A Interpretability Analysis of Additive Rule Methods

Additive rule methods remain readable at the individual-rule level but can impose substantial cognitive work when tracing a single prediction. The analysis identifies overlapping effects, nested thresholds, and activation multiplicity as recurring sources of model-level interpretability burden, while RPTE avoids these patterns through source-disjoint trees and one active leaf per tree.

  • Interpretability motivation: Individual rules in RuleFit, FIGS, and SIRUS are human-readable, but their fitted collections can make reconstructing a single prediction cognitively burdensome.The concern is operational: the mental work required to trace a prediction, rather than a mathematical error.
  • Conflicting overlapping effects: Overlapping rules on the same features can fire with opposite signs, preventing any single rule from expressing the model’s conclusion about those features.Conflicting contributions make rule-level explanations difficult to combine into one clear interpretation.
  • Nested threshold families: Nested threshold rules accumulate when one rule’s coverage is a subset of another’s, leaving no single clear decision boundary.The issue arises when multiple rules use the same variables at different thresholds.
  • Activation multiplicity: The number of simultaneously firing rules per observation determines the cognitive load of tracing its prediction.More active rules require a reviewer to integrate more contributions.
  • RPTE contrast: RPTE separates source variables across trees and activates exactly one leaf within each tree, while its strict variant also constrains retained direct terms.Shared variables can still produce meaningfully distinct threshold, direct, and interaction effects; the concern is repeated overlapping or nested subgroup descriptions.

B Hyperparameter Configuration

Hyperparameters are selected through nested cross-validation, with eight candidate configurations evaluated per model and the best configuration refitted on the full outer-training fold.

  • Hyperparameter search: All models use inner three-fold stratified cross-validation within each outer-training fold for hyperparameter tuning.This nested procedure evaluates candidates using mean inner-fold ROC AUC.
  • Hyperparameter search: Eight candidate configurations are evaluated for every model.The eight configurations arise from the Cartesian product of three binary choices.
  • Hyperparameter search: The configuration with the highest mean inner-fold ROC AUC is refitted on the complete outer-training fold.Table 16 summarizes the eight-configuration search spaces evaluated by inner three-fold stratified cross-validation.

C Detailed Ablation Results

The ablation results show that HUG-based representations substantially improve compactness while preserving comparable predictive performance. HUG-no-augmentation minimizes inspection complexity on nearly all datasets, whereas full RPTE most often maximizes AUC.

  • Ablation results: 11 of 12 datasets have the lowest MIU under HUG-no-augmentation.All entries are means over the 15 outer splits of the 3 × 5 repeated-CV protocol.
  • Ablation results: Full RPTE achieves the highest AUC on seven datasets, including a tie with frequent RPTE and no reassignment on corral.
  • Ablation results: The frequent vocabulary has higher MIU than both HUG configurations on 11 datasets.Overall, HUG-based representations are substantially more compact while retaining comparable predictive performance.
Loading 2608.15725v1…