Hide&Seek: Learning to Explain in an End-to-End Differentiable Network

Introduction and Context

• Feature selection supports interpretation and can simplify models, while instance-wise selection addresses cases where important features vary across individuals.Earlier global methods identify features with high predictive contribution across an entire dataset.Source: §1 · p. 1

Related Work

• L2X fixes subset size across instances, INVASE trains slowly, and REAL-x avoids leakage through decoupling but sacrifices end-to-end differentiability and requires an expressive predictor (Chen et al., 2018; Yoon et al., 2018; Jethani et al., 2021).The predictor must handle arbitrary feature subsets, whose number grows exponentially with data dimensionality.Source: §1 · p. 2

Information Leakage

• Fixed ablation values can leak information because predictors may exploit ablation patterns; for switch features, this can suppress genuinely important features from selection.A switch feature’s induced partition, rather than its exact value, determines the response.Source: §1 · p. 2

Contributions

• Hide&Seek jointly trains a selector and predictor end-to-end by proportionally replacing features, using feature-distribution draws to prevent leakage, and annealing parsimony to stabilize training.The schedule emphasizes predictive performance early in training and increases parsimony later.Source: §1 · pp. 2–3
• The paper generalizes leakage analysis, shows the usual loss can favor omitting important features, and derives a lower bound on feature-importance misidentification that observed rates often approach (Jethani et al., 2021).The lower bound is derived in Corollary A.2.Source: §1 · pp. 2–3

2. Problem Formulation

• Instance-wise feature selection seeks a minimal subset S of features for each input x that is important for predicting its label y.The feature space has d dimensions, with feature indices D = {1, . . . , d}.Source: §2 · p. 3
• Because the selected vector xS varies in dimensionality, a common fixed-size representation replaces unselected features with uninformative values, but this approach requires discrete selection and can leak information.A jointly trained predictor may exploit fixed ablation values, while discrete feature removal is non-differentiable.Source: §2 · p. 3
• A continuous mask m combines each original feature xj with a stochastic replacement value, where larger mj indicates a greater likelihood that feature j is selected.Rather than replacing whole features, the mechanism replaces a proportion of each feature.Source: §2 · pp. 3–4
• The relaxed selection task is to find the most parsimonious real-valued mask m in [0, 1]^d.The continuous masks allow each feature to be partly selected and partly replaced.Source: §2 · p. 4
• Replacement drawn from the original feature distribution makes original and replaced signals indistinguishable to the predictor, but accurately sampling the ideal conditional distribution is non-trivial.In practice, the paper samples replacements from the product of marginal feature distributions.Source: §2 · p. 4
• Marginal replacement is efficient and harder to exploit than fixed-value ablation, and experiments report no leakage under this strategy even in highly correlated settings.The predictor would need to learn the true joint data distribution and identify when stochastic replacements are out of distribution.Source: §2 · p. 4

2.1. Optimization

• The optimization relaxes the selection objective using KL divergence, approximates the unknown conditional with the model-induced distribution, and adds an l1 penalty to encourage sparse selections.The regularization weight balances KL minimization against sparsity; for classification, the negative log-likelihood is standard cross-entropy.Source: §2.1 · p. 4

3. Proposed Model

• Hide and Seek jointly learn feature importance and predictions using continuous masks that replace a proportion of each feature with marginal noise, avoiding discrete feature sampling.Hide produces a continuous importance mask, and Seek maps the resulting input to a predicted output.Source: §3 · pp. 4–5

3.1. Parsimony-Weight Annealing

• Quadratically increasing the parsimony weight toward its maximum prioritizes prediction accuracy early and mask parsimony later, leading to lower final loss and strong feature-importance metrics.Excessive emphasis on parsimony early in training can trap optimization in poor local minima.Source: §3.1 · p. 5

4.1. Synthetic Data

• Hide&Seek outperforms the other models on synthetic instance-wise feature-selection metrics and achieves higher predictive performance with 17×–30× fewer parameters than other selector–predictor models.It is also described as easy to train because it is end-to-end differentiable; INVASE and REAL-x have significantly longer training times. Synthetic feature recovery is evaluated with TPR and FDR across global and instance-wise settings.Source: Table 2 · p. 6, §4.1 · p. 6

4.3. Correlated Data

• Under multicollinearity, Hide&Seek consistently outperforms other models on instance-wise feature selection and maintains switch accuracy above 99.4% averaged across Syn4–6 for every tested ρ.The experiment varies the pairwise feature correlation ρ from 0 to 0.9; the reported switch accuracy indicates no leakage even when the switch feature is highly correlated with other features.Source: Figure 4 · p. 7, §4.3 · p. 7

4.4. Credit Default Data

• The credit-default experiments assess feature selection with correlated real-world attributes and prediction on synthetic-label settings and actual default labels, without reporting comparative outcomes in the supplied passages.The dataset has 30,000 instances and 23 features; AGE serves as the switch feature, and prediction is evaluated with mean AUROC across 20 runs.Source: §4.4 · p. 7, Table 4 · p. 8, Table 5 · p. 8

4.5. MNIST

• At 99.1% or greater classification accuracy, only Hide&Seek and REAL-x consistently identify the discriminative left arcs, while LIME misses important patches and INVASE and SHAP show less varied explanations.The comparison uses the four most important 3 × 3 patches per image for distinguishing handwritten 3s from 8s.Source: Figure 5 · p. 8, §4.5 · p. 8

4.6. Breast Cancer Subtype Classification

• On BRCA microarray data, Hide&Seek ranked ESR1 and CCNB2 substantially above the other genes; both have established BRCA associations and were identified by Covert et al. (2021).The analysis ranks 100 selected genes using mean mask size over a test set of 100 instances and highlights candidate gene associations beyond those previously reported.Source: §4.6 · pp. 8–9, Table 6 · p. 9

5. Conclusion

• Hide&Seek jointly trains a selector and predictor, replacing unselected features with distributional draws and using continuous masks and parsimony-weight annealing to stabilize training.Distributional replacement prevents the predictor from exploiting the ablation pattern; annealing prioritizes predictive accuracy early and sparsity later.Source: §5 · p. 9
• The usual joint ablation-model objective can be an ill-posed proxy for the intended expected KL objective, causing important features to be omitted and bounding switch-feature identification.Experiments found leakage-prone methods often near the theoretical misidentification bound, while Hide&Seek maintained high switch accuracy.Source: §5 · p. 9
• With binary masks and replacement values conditioned on selected features, Hide&Seek makes information leakage theoretically impossible; its continuous-mask relaxation with marginal feature draws remains empirically robust.The idealized construction uses conditional replacement, while the empirical variant draws from marginal feature distributions.Source: §5 · p. 9
• Hide&Seek achieves strong instance-wise feature-selection performance across synthetic, credit-default, MNIST and genetic microarray data while remaining fast to train.The results suggest distributional replacement avoids an artificial information channel while preserving joint selector–predictor training.Source: §5 · p. 9

• Improved interpretability of black-box models can support transparency and informed decision-making, but misleading explanations pose risks and require careful validation.Source: Impact Statement · p. 9
• The paper proves that a seemingly natural loss used in prior XAI work is ill-posed because it can cause unintended information leakage, highlighting a broader objective-design challenge.Source: Impact Statement · p. 10

A. On Feature Ablation and Information Leakage

• Ablation converts conditioning on an arbitrary selected feature subset into a fixed-size vector by replacing unselected features with a symbol outside their original support.Source: A. On Feature Ablation and Information… · p. 12
• Ablation-based proxy losses can favor selector–predictor pairs that exploit the mask, so their minimizers need not minimize the intended loss.Under the theorem’s partition-dependence assumption, a constructed pair can lower proxy loss by ablating a feature on one partition cell while preserving the intended prediction; elsewhere its loss contribution is equal.Source: A. On Feature Ablation and Information… · pp. 12–14
• The achievable misidentification rate is bounded below by the probability mass of the largest partition cell, but this is an existence guarantee, not a rate every training run attains.Other leakage strategies may achieve still higher rates.Source: A. On Feature Ablation and Information… · p. 14

A.1. Comparison to Jethani et al. (2021)

• Jethani et al. (2021) identified leakage in two JAM scenarios: class-label encoding in the binary mask and omission of tree control-flow features when selected leaf features reveal the active branch.Source: §A.1 · p. 14
• The analysis extends leakage to switch features, where the output depends on a feature’s partition rather than its exact value, in both regression and classification settings.Unlike the earlier scenarios, the relaxed setting has no restriction on the number of classification labels or, for tree-generated outputs, on feature sets across leaves.Source: §A.1 · pp. 14–15

A.2. Sampling ˆx Using Model-X Knockoffs

• Marginal sampling outperforms Model-X knockoff sampling in the reported comparisons, while conditional replacement sampling is described as a stronger theoretical guarantee against leakage.Hide&Seek draws replacement values from the product of marginals; the text points to Sections 4.2, 4.3, and A.4 for evidence of no leakage in that setting.Source: Figure 6 · p. 15, §A.2 · p. 15
• Model-X knockoff exchangeability for a fixed subset does not carry over directly because the selector chooses the masked subset as a function of the data.Source: §A.2 · p. 16

A.3. Preventing Information Leakage

• If the predictor’s modified input has the original feature distribution, it cannot distinguish modified from original inputs; for binary masks, this requires drawing unselected replacements from p(X_¯S | x_S).The section presents this conditional distribution as the ideal replacement rule for preventing possible leakage.Source: §A.3 · p. 16

A.4. Correlated Synthetic Data

• Even with marginal sampling, Hide&Seek could not learn that replacement values for the switch feature were out of distribution; Table 7 reports switch accuracy across ρ and models.The table reports mean accuracy identifying X11 as important across Syn4, Syn5, and Syn6 for one run per model and ρ value; no cell values are provided here.Source: §A.4 · p. 16, Table 7 · p. 16

B. Model infrastructure

• Hide&Seek uses two feed-forward networks with two 32-unit hidden layers each, trained for 500 epochs without batching using Adam at learning rate 0.001.Hide’s output is sigmoid-constrained to [0, 1]^d and Seek’s is softmax; training shuffles data columns with replacement each epoch to create replacement values.Source: B. Model infrastructure · p. 17
• INVASE and REAL-x use 100-unit selector and 200-unit predictor layers; they train for 10,000 and 500 epochs, respectively, with λ=0.1 and λ=0.15.Both use batch size 1,000 for the Section 4.1 synthetic data.Source: B. Model infrastructure · p. 17
• L2X uses two networks with two hidden layers each, sized 100 units in the first network and 200 in the second.The implementation uses the L2X codebase.Source: B. Model infrastructure · p. 17
• LIME uses its cited implementation, and the paper’s repository contains the baseline models for Synthetic and MNIST data.Source: B. Model infrastructure · p. 17
• For SHAP, the authors use TreeExplainer with an XGBoost predictor because it performed better than KernelExplainer on synthetic data; the tuned model uses depth 5 and 100 boosting rounds.The stated XGBoost settings also include binary:logistic objective, logloss evaluation, eta 0.1, and colsample_bytree 0.9.Source: B. Model infrastructure · p. 17
• The RForest baseline uses Gini criterion, 100 trees, and maximum depth 5; LASSO is implemented as logistic regression with an L1 penalty.Source: B. Model infrastructure · p. 17

B.1. Hardware

• Experiments ran on an AMD EPYC 9354P CPU, 192GB DDR5 RAM, two 1.92TB NVMe SSD drives, and two NVIDIA L4 GPUs.The listed GPUs have 7,680 cores, 240 Tensor Cores, and 24GB memory each.Source: §B.1 · p. 17

C. Metric calculations

• TPR, FDR, and F1 were calculated per input instance and averaged across the dataset; where specified, 20 seeded runs were summarized by their median.Figure 7 shows distributions for the Section 4.1 feature-identification scores across 20 experiments.Source: C. Metric calculations · p. 18, Figure 7 · p. 18
• For each instance, TPR is true positives divided by true positives plus false negatives, while FDR is false positives divided by true positives plus false positives.Recall equals TPR, and precision equals 1 − FDR.Source: C. Metric calculations · p. 18

D. Further analyses

• Section D presents further analyses and experiments concerning the synthetic-data experiment in Section 4.1.Source: D. Further analyses · p. 19

D.1. Mask Distributions

• Hide&Seek’s learned mask distributions closely align with the data-generating rules across the six synthetic experiments.Figure 8 shows histograms and KDE plots for one of the 20 experiments; X11 is the switch-feature in Syn4–6.Source: §D.1 · p. 19, Figure 8 · p. 19

D.2. Predictive Performance

• Hide&Seek consistently achieves higher AUROC than INVASE, L2X, and REAL-x despite having significantly fewer parameters.Figure 9 presents predictive performance for the models used in Section 4.1.Source: Figure 9 · p. 20, §D.2 · p. 20
• REAL-x’s poorer predictive performance may reflect out-of-distribution draws when predicting on highly parsimonious data.Its predictor is trained separately using feature masks drawn from a Bernoulli(0.5) distribution.Source: §D.2 · p. 20

D.3. λ Sensitivity

• Across λ settings, Hide&Seek keeps feature significance near ground truth and almost all AUROC values above 0.8, while INVASE and REAL-x become overly parsimonious.REAL-x also quickly loses prediction accuracy as λ increases, potentially because its predictor is trained on Bernoulli(0.5) masks and encounters OOD shift for highly parsimonious inputs.Source: §D.3 · p. 20, Figure 10 · p. 21
• Hide&Seek’s AUROC strongly correlates with IWFS F1, initially rising with parsimony, whereas INVASE can show high prediction accuracy alongside low F1.REAL-x’s highest AUROC occurs with the most features available; AUPRC-versus-λ plots show similar patterns.Source: §D.3 · p. 20
• INVASE used λ = 0.1 for the reported results; choosing λ = 0.95 or λ = 0.45 for highest AUROC would have lowered its IWFS results.The λ = 0.1 value matches the original INVASE paper’s synthetic-data experiments and corresponds to high IWFS F1.Source: §D.3 · p. 21

D.4. Results for k = 3 and k = 4

• For Syn4–5, the main-text top-k setting may overestimate FDR, so SHAP, LIME, and L2X results are also shown for k = 3 and k = 4.The main-text settings use k = 5 for Syn4–6; Syn4 and Syn5 have only 3 important features when X11 < 0. Table 8 reports medians over 20 experiments.Source: §D.4 · p. 22, Table 8 · p. 22

D.5. Run Time Analysis

• Hide&Seek uses approximately 3k parameters, versus approximately 57k for REAL-x and 100k for INVASE; INVASE training is substantially longer than other methods.REAL-x uses REBAR gradients (Tucker et al., 2017) but trains selector and predictor networks separately; Hide&Seek does not use batching, unlike REAL-x, INVASE, and L2X. Table 9 reports typical synthetic-data run times.Source: §D.5 · p. 22, Table 9 · p. 22

D.6. Syn3 - Specification vs. Implementation

• For comparable Syn3 experiments, the authors use the data-generating model from the previous code because it differs from the specification in prior works (Chen et al., 2018).Table 10 contrasts the paper and code expressions for Syn3 in INVASE, L2X, and Hide&Seek.Source: §D.6 · p. 22, Table 10 · p. 22

D.7. Parsimony-Weight Annealing Analysis

• As q increases at fixed λmax, fewer epochs use larger λt values, resulting in less parsimony.The q=2 schedule allows prediction accuracy to be emphasized early and parsimony later in training.Source: §D.7 · p. 23
• The annealing analysis concerns the λt schedule over t epochs, whose definition is outlined in section 3.1.The experiments use q=2.Source: §D.7 · p. 23
• For q ∈ {0.5, 1}, results are poor, while q ∈ {2, 3, 4, 5} yields stable feature-selection and prediction results.The reported metrics are TPR, FDR, F1, and AUROC.Source: Table 11 · p. 23, §D.7 · p. 23
• Table 11 reports mean TPR, FDR, F1, and AUROC across six synthetic datasets and 20 seeds, with λmax=0.3 for every run.TPR, FDR, and F1 measure instance-wise feature selection; AUROC measures prediction.Source: Table 11 · p. 23
• With λmax large enough to impose parsimony, q=2 gives the best results, is more stable than q=3, and yields a lower final combined loss.At t=500, the loss is the same for all four tested q values and is comparable.Source: §D.7 · p. 23, Figure 11 · p. 24

E. Further experiments

• Further experiments examine training on 1,000,000 samples, performance with 100 features, and California Housing data.Source: E. Further experiments · p. 25

E.1. Training on 1,000,000 Samples

• With 1,000,000 training samples, Hide&Seek still outperforms the other models, while L2X improves with more data.Table 12 reports results from a single experiment per model and dataset.Source: §E.1 · p. 25, Table 12 · p. 25
• As training data grew from 10,000 to 1,000,000 samples, REAL-x required a lower parsimony regularizer λ, unlike Hide&Seek and INVASE.Source: §E.1 · p. 25

E.2. Training on 100 Features and Ensembling

• The 100-feature synthetic experiment added 89 independent noise signals to the original 11 features, and Hide&Seekens ensembled 10 models trained on random 90% feature subsets.Feature importance was obtained by averaging ensemble masks and applying the standard > 0.5 selection threshold.Source: §E.2 · p. 25
• Hide&Seekens outperformed all other models and matched or exceeded ideal INVASE, while base Hide&Seek was competitive.INVASEideal denotes performance when its ideal λ was known; Table 13 reports median F1 across 20 experiments.Source: §E.2 · p. 25, Table 13 · p. 25

E.3. California Housing

• The California Housing semi-synthetic experiment used seven census variables and a Longitude switch feature with three geographic partitions to define ground-truth importance.The target was sampled from a Bernoulli distribution whose probability depended on U; the supplied partition rules specify U for Longitude < −121.5 and Longitude ≥ −118.Source: §E.3 · p. 26
• The California Housing evaluation tested recovery of housing attributes and Longitude, reporting median TPR, FDR, and F1 over 10 runs on the same test data.The data split was 12,828 training, 1,604 validation, and 1,604 test samples; features were standardized using training data, and λ was tuned for Hide&Seek, REAL-x, and INVASE.Source: Table 14 · p. 26, §E.3 · p. 26

F. Extra detail

• The final supplementary section provides further detail on the paper’s non-synthetic data experiments.Source: F. Extra detail · p. 26

F.1. Credit Default Data Detail

• The Credit default experiment used 23 input features, with 11 designated to generate Syn4–6.The raw-data target is default payment next month and is not used in Syn4–6; the feature list includes credit limit, bill and payment amounts, demographics, and payment-status variables.Source: §F.1 · p. 26, Figure 12 · p. 27

F.2. MNIST Detail

• MNIST hyperparameter λ was tuned over six values for Hide&Seek, INVASE, and REAL-x, with settings selected by prediction accuracy.The tested values were 0.05, 0.1, 0.2, 0.3, 0.4, and 0.5.Source: §F.2 · p. 26
• For Hide&Seek on MNIST, training-set standardization yielded higher accuracy than global [0, 1] rescaling at every tested λ.The selected λ was 0.05, although explanation patches were largely similar across all six values.Source: §F.2 · p. 26

F.3. Breast Cancer Subtype Classification Detail

• The breast cancer subtype analysis used 100 genes to match the experiment in Covert et al. (2021).Table 15 lists the genes used.Source: §F.3 · p. 26, Table 15 · p. 27
• Table 6 lists each model’s top 10 genes ranked by mean importance, and Table 16 gives detailed importance values with standard errors.Importance is measured as mean mask size for Hide&Seek, INVASE, and REAL-x; mean importance scores for SHAP and LIME; or mean selection frequency for L2X.Source: §F.3 · pp. 26–27, Table 16 · p. 27

F.1. Credit Default Data Detail

• Figure 12 presents the correlation matrix for the 23 Credit default input features.The 11 features used as X1–X11 to generate Syn4–6 are outlined in black.Source: Figure 12 · p. 27