Look at the Variance! Efficient Black-box Explanations with Sobol-based Sensitivity Analysis

1 Introduction

• White-box methods require access to internal states, limiting their use when models are exposed through third-party APIs or specialized hardware.Black-box methods instead use the model’s forward-pass predictions.Source: §1 · pp. 1–2
• Existing black-box methods face costly Monte Carlo sampling and simple perturbations that constrain exploration of the input space.Single-location perturbations miss nonlinear interactions, while assessing combinations quickly becomes combinatorially intractable.Source: §1 · p. 2
• Sobol indices with perturbation masks capture both individual region importance and higher-order interactions in black-box predictions.The method supports real-valued intensity perturbations and uses Quasi-Monte Carlo sampling with efficient estimators for high-dimensional inputs.Source: §1 · pp. 2–3
• At least half the forward passes yield explanations on par with or better than state-of-the-art methods.The paper evaluates image classification on Pointing Game and Deletion benchmarks and also reports results for text classification.Source: §1 · p. 3

2 Related work

• Gradient- and activation-based approaches require internal computations, whereas the proposed method is model-agnostic.Iterative mask optimization can take 20 seconds per explanation; the proposed approach avoids iterative gradient computation and can be parallelized.Source: §2 · p. 3
• Occlusion perturbs one region at a time and therefore misses nonlinear interactions between regions.Occluding two related facial features separately may have little effect even when occluding them together substantially changes the prediction.Source: §2 · p. 3
• Unlike binary-mask approaches such as RISE, the method supports continuous perturbations and a theoretical decomposition of region influence across orders.It can also use perturbations from Occlusion, LIME, and RISE, as well as perturbation functions that accept continuous inputs.Source: §2 · pp. 3–4

3 Sobol attribution method

• The method estimates Sobol indices by perturbing real-valued features and measuring their effect on black-box prediction scores.The perturbation distribution defines the space of possible inputs; inpainting and blurring are examples for images.Source: §3 · p. 4, §3.1 · p. 4

3.2 Sensitivity analysis using Sobol indices

• First-order indices measure a feature’s intrinsic share of output variance, while total indices include its interactions of any order.A large difference between total and first-order indices indicates stronger interactions with other variables.Source: §3.2 · pp. 5–6

3.3 Efficient estimator

• The Jansen estimator is paired with Quasi-Monte Carlo sampling to support efficient, stable convergence.QMC uses low-discrepancy sequences, in place of the Monte Carlo sampling typically used with Jansen.Source: §3.3 · p. 6
• Estimating first-order and total indices requires N(d+2) model forward passes.The procedure draws two N × d perturbation-mask matrices and forms a matrix that replaces one variable’s column at a time.Source: §3.3 · p. 6

3.3.1 Signed estimator

• The signed variant combines total Sobol importance with the sign of the prediction-score change caused by occluding a feature.The score difference compares the original input with a partial version in which that feature is occluded.Source: §3.3.1 · p. 6

4.1 Pointing game

• The proposed Sobol method outperforms RISE in all tested cases while reducing forward passes by half; the signed estimator leads on Pascal VOC/VGG16, and both estimators prevail on COCO/VGG16.Source: §4.1 · p. 8

4.2 Fidelity

• Sobol’s total estimator is the most faithful tested black-box method on ImageNet Deletion and matches Integrated Gradients overall, scoring 0.121 versus 0.123 on ResNet50V2.Deletion evaluates how quickly the prediction score drops as pixels ranked important are reset; the authors use it alongside plausibility metrics because those can fail on incorrect predictions or assume human-like evidence.Source: §4.2 · p. 8, Table 2 · p. 9

4.3 Efficiency

• 0.8 vs 0.65 Spearman rank correlation after 1,000 forwards, and 0.97 vs 0.73 after 7,000, favor Sobol over RISE.Sobol also scores 0.151 on Deletion with 492 forwards versus 0.121 with 3,936; the convergence experiment compares explanations with 10,000-forward references using rank correlation.Source: Figure 2 · p. 9, §4.3 · p. 9

4.4 Word deletion

• 0.461 Word deletion for Sobol S_delta Ti on bi-LSTM beats the best tested white-box score of 0.529, but Occlusion is more faithful overall on both tested models.Occlusion scores 0.440 on LSTM and 0.531 on BERT, versus Sobol S_delta Ti scores of 0.461 and 0.598; lowering the mask threshold to 0.05 yields 0.527 for BERT.Source: Table 3 · p. 10, §4.4 · p. 10

5 Conclusion

• Sobol attribution uses perturbation masks and efficient estimators to make variance-based sensitivity analysis tractable for high-dimensional data and capture higher-order interactions.The paper reports performance on par with or better than leading black-box methods while being twice as fast, and comparable results to leading white-box methods without access to internal states.Source: §5 · p. 10
• The method also produced initial competitive results on language models, alongside its demonstrated use for explaining image classifiers.Source: §5 · p. 10

A Broader Impact

• Explanations should not be trusted blindly: the attribution methods are sensitive to adversarial attacks that can hide model behavior, an open problem for attribution methods.Source: A Broader Impact · p. 15
• The authors frame explanations as a necessary first step toward correcting biases in deployed AI systems, not as a guarantee that those systems are unbiased.Source: A Broader Impact · p. 15

B Qualitative comparison

• STi consistently produces sparser maps than RISE while remaining equally consistent.Source: B Qualitative comparison · p. 15
• Gradient-based methods generally produce the sharpest maps, but some are prone to failure.The paper points to an example in the fourth row of Figure S1 and notes that this failure is known.Source: B Qualitative comparison · p. 15
• The comparison uses the first ImageNet validation images rather than hand-picked examples; gradient-based maps were clipped at the 2nd percentile for visualization.Source: B Qualitative comparison · p. 15

C Effectiveness of modeling higher-order interactions

• Sobol STi and Sobol signed combine first-order and higher-order interactions between image regions.Source: C Effectiveness of modeling… · p. 15
• Occlusion considers one region at a time, whereas RISE removes around 50% of regions at a time; the section notes that RISE already surpasses Occlusion on ImageNet Deletion scores.Source: C Effectiveness of modeling… · p. 15
• 0.121 versus 0.170 Deletion scores on ResNet50v2 favor Sobol total indices over first-order indices; similar differences appear on VGG16, EfficientNet, and MobileNetV2.Lower scores are better.Source: C Effectiveness of modeling… · p. 15, Table S1 · p. 15

D Efficiency of Sobol estimator

• A brute-force estimator derived from the Sobol-index definition can be computationally costly, particularly for total Sobol indices and complex neural networks.Source: D Efficiency of Sobol estimator · pp. 15–16
• Sobol sequences provide faster convergence than ordinary Monte Carlo for sufficiently regular functions, and the estimator consistently converges faster than RISE in the reported figures.The sequence covers the integration hypercube more uniformly.Source: D Efficiency of Sobol estimator · p. 16
• 0.151 Deletion score with 492 forwards versus 0.121 with the default 3,936 forwards shows a competitive score at a lower forward count.Source: D Efficiency of Sobol estimator · p. 16
• Figures S2 and S3 plot Spearman rank correlation against the number of forwards, relative to a 10,000-forward explanation, for ResNet50V2 and VGG16, respectively.Source: Figure S2 · p. 17, Figure S3 · p. 17

• The methods assign input-feature attribution scores, while f(x) denotes the logit before softmax for the class of interest.Source: E Explanation methods · p. 17
• Gradient-based alternatives include local input gradients, Gradient ⊙Input, path-integrated gradients, and SmoothGrad’s averaged gradients at perturbed inputs.Integrated Gradients uses a zero baseline and 80 trapezoidal steps; SmoothGrad uses 80 samples and σ = 0.2.Source: E Explanation methods · pp. 17–18
• Figure S4 plots explanation rank correlation against forward-pass count relative to a 1,0000-forward explanation; Table S2 covers ImageNet deletion scores for ResNet50V2 with Sobol indices.Lower deletion scores are better in Table S2.Source: Figure S4 · p. 18, Table S2 · p. 18
• Grad-CAM weights last-convolution-layer feature maps for a class, then bilinearly resizes the explanation to the input dimensions.Source: E Explanation methods · pp. 18–19
• Occlusion sweeps an image patch and uses prediction changes to locate critical areas; experiments used patch size and stride of 20.Source: E Explanation methods · p. 19
• RISE probes randomly masked images, using masks sampled in a subspace and bilinearly upsampled; experiments used N = 8,000 and E(M) = 0.5.After upsampling, the masks are no longer binary.Source: E Explanation methods · p. 19

• Insertion assumes faithful explanations make prediction scores rise faster as important original-image pixels are added to a baseline image.Source: F Fidelity with Insertion · p. 19
• Insertion and Deletion were introduced together, but the paper focuses on Deletion in the main text because Insertion is less common in the literature.Source: F Fidelity with Insertion · p. 19
• Table S3 reports insertion scores on 2,000 ImageNet validation images, with Random inserting among the pixels remaining at each step.Higher scores are better; the first- and second-best results are bolded and underlined.Source: Table S3 · p. 19

• The check follows by progressively resetting weights in Inception V3, including the upper layers from the logits.Source: G Sanity check · pp. 19–20
• Figure S5 depicts reinitialization progressing from Mixed 6 to Mixed 1 in InceptionV3 [68].Source: Figure S5 · p. 20
• The STi explanation passes the sanity check, showing sensitivity to modifications of the model weights.Source: G Sanity check · p. 20, Figure S5 · p. 20

• 89% test accuracy was reached by the bidirectional LSTM with 300-dimensional GloVe embeddings and a hidden size of 64.The 64-dimensional hidden size is split into 32 dimensions per direction.Source: H Word Deletion · p. 20
• 92% test accuracy was reached by the BERT-based model, using bert-base-uncased with its final layer tuned by cross-entropy and Adam.The initial learning rate was 1e−3.Source: H Word Deletion · p. 20
• Reducing the average percentage of words removed yields a better deletion score in the reported threshold experiment.Source: H Word Deletion · p. 20
• Table S4 reports BERT-based word-deletion scores while varying the perturbation threshold to control average word presence; lower is better.Source: Table S4 · p. 20