Robust to Which Model Change? A Unified Evaluation of Robust Counterfactual Explanations

• A counterfactual explanation describes feature changes that would make a model produce a target outcome, and may be judged by proximity, plausibility, sparsity, or actionability.Source: §1 · p. 1
• A model retrained or updated before a recommendation is implemented can make the same counterfactual invalid even when the person followed it.Source: §1 · p. 1
• Existing methods choose different model changes for evaluation, so their reported robustness scores answer different questions and cannot be compared directly.Source: §1 · pp. 1–2
• The protocol tests six robust methods and two baselines against the same held-out models from eight change families across four tabular datasets.Source: §1 · p. 2
• The benchmark finds that rankings vary by change family, parameter-perturbation guarantees may not transfer to retraining, and robustness scores can overstate performance without coverage and distance.Source: §1 · p. 2

2.1 Problem setting

• A counterfactual is valid when the base classifier assigns it the target class, and survives a model change when the replacement classifier still assigns that class.Source: §2.1 · p. 2
• For each change family, robustness is the probability of remaining valid under its classifier distribution; empirical robustness averages this probability over returned, base-valid counterfactuals.Source: §2.1 · p. 2
• Coverage and base validity are reported separately from empirical robustness, so generation failures are not concealed by the robustness score.Source: §2.1 · p. 2

2.2 Existing robustness assumptions

• Methods encode future changes differently: from bounded parameter shifts and local stability to distributional ambiguity, sampled model spaces, or neural-network parameter certificates.Source: §2.2 · pp. 2–3, Table 1 · p. 3

2.3 Why a unified evaluation is needed

• Published scores are not comparable because methods were tested on different author-chosen changes, often with different numbers of models and treatment of failed generations.Source: §2.3 · p. 3
• The protocol holds generated counterfactuals fixed across held-out classifiers from all eight families and reports each family separately; failures outside a certificate’s scope do not contradict that guarantee.Source: §2.3 · p. 3

3 Evaluation design

• Each method generates one counterfactual per selected factual for each base model, then the same counterfactual is evaluated on held-out variants not used for generation or tuning.Source: §3 · p. 4

3.1 Datasets and base models

• The benchmark uses four binary tabular datasets, shared stratified splits, and five independent neural-network initializations with the same architecture across methods.Source: §3.1 · p. 4

3.2 Model variants

• Each base classifier has 25 variants in each of eight change families, and variants are characterized by prediction disagreement and probability differences from the base model.Source: §3.2 · p. 4, Table 2 · p. 5

3.3 Counterfactual methods

• Method configurations avoid access to changed evaluation models; BetaRCE uses a separate generation ensemble, while AP∆S uses separate calibration models.Source: §3.3 · p. 4

3.4 Metrics and uncertainty

• Coverage, base validity, empirical robustness, and proximity are reported separately, with robustness averaged over base-valid counterfactuals and proximity scaled by training-feature variability.Source: §3.4 · p. 5
• Metrics are averaged over five independently trained base models, with standard deviations across those models because the base model is the replication unit.Source: §3.4 · p. 5

4.1 Robustness depends on the model-change family

• Bootstrap retraining changes 4.92% of hard test predictions, versus 0.95% for bounded parameter perturbations; architecture changes alter 4.69%.The parameter-robust methods’ uncertainty sets cover the family that changes predictions least, limiting what their guarantees say about retraining or replacement.Source: §4.1 · p. 5
• The two RobX configurations never fall below 99.4% empirical robustness, while BetaRCE stays above 91% and RNCE never below 95% across dataset–family combinations.Source: §4.1 · p. 5
• There is no stable method ranking across change families: AP∆S reaches 94% under bounded perturbations but only 11% under new initialization and 16% under architecture changes on Breast Cancer.Figure 1 shows empirical robustness by dataset, method, and change family.Source: §4.1 · p. 5, Figure 1 · p. 6

4.3 Robustness has a price in distance

• On every dataset, the closest and most robust methods differ; RobX robust-first is most robust and farthest, while AP∆S produces the closest CFEs.The nearest-neighbor baseline is 12 to 18 points less robust than robust methods on three datasets, despite moving as far.Source: §4.3 · p. 6
• RobX’s stricter threshold raises distance on every dataset, including from 0.959 to 1.367 on HELOC, although both configurations are equally robust.Both configurations anchor counterfactuals at a stable training instance.Source: §4.3 · p. 6

4.4 Formal guarantees retain their stated scope

• AP∆S empirical robustness is 94.1% to 100.0% under bounded perturbations, versus 58.5% on Breast Cancer, 83.5% on Diabetes, 95.7% on Wine Quality, and 94.7% on HELOC across families.Source: §4.4 · pp. 6–7

4.2 Robustness must be read with coverage and validity

• On Wine Quality, ROAR-LIME’s 93.1% empirical robustness falls to 67.8% end-to-end because only 72.8% of its candidates are base-valid.Figure 2 relates coverage, empirical robustness, and end-to-end robustness.Source: §4.2 · p. 6, Figure 2 · p. 7

4.4 Formal guarantees retain their stated scope

• RNCE’s CFEs remain valid for 97.9% to 99.2% of retrained and re-architected models in this benchmark, but that transfer is a property of these procedures, not its certificate.A different retraining procedure could invalidate many of the same CFEs without contradicting the bounded-parameter guarantee.Source: §4.4 · p. 7

5 Discussion

• Robustness preference depends on the update family and acceptable intervention size; similar disagreement rates can still invalidate different counterfactuals.Output disagreement describes a change distribution but does not replace evaluating each family separately.Source: §5 · p. 7

5.1 A unified evaluation protocol for robust counterfactuals

• Competing methods should be tested against the same held-out model changes, described by hard disagreement and continuous prediction differences.Parameter distance is useful only when parameters are aligned, not for comparing architectures.Source: §5.1 · p. 7
• Coverage, base validity, and empirical robustness answer distinct questions; report end-to-end robustness alongside them and pair the rates with proximity.Robustness may require a larger intervention.Source: §5.1 · p. 7
• Figure 3 aggregates five seeds per point and traces a proximity–robustness frontier among base-valid CFEs.Both plotted quantities are evaluated only among base-valid counterfactuals.Source: Figure 3 · p. 8
• The experimental replicate is the independently trained base model, not each CFE–variant pair.Seed-wise means and standard deviations should reflect that hierarchy.Source: §5.1 · pp. 7–8

5.2 Limitations

• The benchmark is limited to binary numeric tabular tasks and differentiable neural networks, and it omits causal feasibility, immutable features, recourse time, sparsity, diversity, and plausibility.RNCE and AP∆S require differentiable networks, which constrains the methods comparable under the same classifiers.Source: §5.2 · p. 8

6 Conclusion

• A counterfactual robust to one kind of model change need not remain robust to another; methods surviving small parameter perturbations can lose counterfactuals after retraining.Methods that hold up under retraining usually require a larger change from the person.Source: §6 · p. 8
• Competing methods should be tested against the same held-out changed models, with results separated by change type and coverage, base validity, robustness, and distance reported separately.The conclusion says this reporting makes results comparable across methods and papers.Source: §6 · p. 8
• The reported results are intended to help practitioners anticipate performance under the update their own model will receive.The conclusion ties that use to reporting results separately for each kind of change.Source: §6 · p. 8

A.1 Datasets, splits, and base classifiers

• All five base-model runs use the same fixed data partition, with adverse test predictions as eligible factuals.Every eligible factual is used for Breast Cancer and Diabetes; Wine Quality and HELOC use reproducible samples of 250 per base model.Source: §A.1 · p. 10
• Table 3 summarizes dataset sizes, feature counts, training/update/validation/test splits, factual counts, and base-model test balanced accuracy.Factual counts include the total adverse test predictions across five models and the per-model range; balanced accuracy is shown as minimum/mean/maximum.Source: Table 3 · p. 10
• Each base classifier has two hidden layers of 32 ReLU units and a sigmoid output, trained with full-batch Adam at learning rate 10^-3 and no weight decay.Five runs differ in initialization while sharing the data partition and training protocol; early stopping uses validation binary cross-entropy with patience 30.Source: §A.1 · p. 10

A.2 Realized magnitude of model changes

• Bounded parameter perturbations use five radii and five random directions per radius, while the training-configuration family varies optimizer settings and excludes the base setting.Directions are scaled so their largest component equals the radius; the training-configuration family has 19 Adam and 6 SGD variants.Source: §A.2 · p. 10
• Table 4 reports output differences as mean ± standard deviation over 125 variants per dataset, macro-averaged across datasets.The table describes realized model-change magnitude relative to the base classifier.Source: Table 4 · p. 10

A.3 Method configuration and separation of model pools

• Wachter and ROAR-LIME tuning prioritizes base-model validity, breaking ties by lower robust-scale ℓ1 distance; ROAR-LIME uses a perturbation radius of 0.1.The validity-first ordering follows the ROAR authors’ selection principle.Source: §A.3 · p. 10
• RNCE uses parameter and bias radii of 0.005, RobX uses variance 0.01 and 1,000 stability samples, and BetaRCE uses 32 bootstrap models with δ = 0.9 and α = 0.95.RobX thresholds are set from target-class training-instance stability; the BetaRCE bootstrap samples differ while initialization is fixed to the base model.Source: §A.3 · p. 11
• AP∆S defines its radius as the maximum parameter ℓ∞ distance among ten bootstrap-update replicas, with calibration replicas separate from held-out evaluation variants.The selected radii ranged from 0.007494 to 0.007743 on Breast Cancer, 0.009563 to 0.009802 on Diabetes, 0.034274 to 0.050010 on Wine Quality, and 0.046518 to 0.049291 on HELOC.Source: §A.3 · p. 11
• AP∆S returned no CFEs in a small independent-initialization diagnostic, showing that its parameter radius is meaningful relative to the process that produced the models.The diagnostic is explicitly outside the main comparison.Source: §A.3 · p. 11

A.4 Generation time

• Table 5 reports per-request generation times across 3,093 requests per method, including unsuccessful attempts but excluding training, calibration, and ensemble construction.The measurements characterize these implementations on the reported hardware, not implementation-independent complexity.Source: §A.4 · p. 11, Table 5 · p. 11

B Full dataset-wise results

• End-to-end robustness is the product of coverage, base validity, and empirical robustness.Empirical robustness is the mean RQg among the method’s base-valid CFEs.Source: B Full dataset-wise results · p. 12
• Distance is the mean robust-scale-normalized ℓ1 change among base-valid counterfactuals.Source: B Full dataset-wise results · p. 12
• Table 6 reports every dataset–method result as a mean and standard deviation over five independently trained base models.Rates are percentages.Source: Table 6 · p. 12, B Full dataset-wise results · p. 12