Active Testing: Sample-Efficient Model Evaluation

1. Introduction

• Active testing produces more precise test-loss estimates than uniform sampling at the same label count, while reducing the labels required for unbiased evaluation.The paper presents this as a sample-efficient evaluation result.Source: Figure 1 · p. 1, §1 · pp. 1–2
• Test labels matter because practitioners use evaluation to choose models and develop trust, yet their cost has largely been ignored in active-labeling research.In scientific applications, a label may require expert researchers and weeks of lab time.Source: §1 · p. 1
• The framework derives principled acquisition strategies for classification and regression, with practical approaches ranging from original-model methods to surrogates that use acquired labels.The strategies depend on predictive models for outputs, allowing customization to problem and computational budgets.Source: §1 · pp. 1–2
• Because model-informed test selection creates sample-selection bias, the paper uses a weighting scheme to remove it.The authors note that selecting least-certain points can overestimate test loss and undermine model selection or hyperparameter optimization.Source: §1 · p. 2
• The approach is presented as applicable to practically any machine-learning model and task, with demonstrations spanning neural networks, Gaussian processes, and random forests.The examples range from toy regression to image classification such as CIFAR-100.Source: §1 · p. 2

2. Active Testing

• Active testing estimates a statistic such as accuracy or loss for a fixed model, which remains unchanged during evaluation.The framework assumes few model restrictions and applies across supervised-learning tasks.Source: §2 · p. 2

2.1. A Naive Baseline

• The naive baseline uniformly samples a test subset, yielding an unbiased estimate that can have high variance when the labeling budget is much smaller than the test set.The estimate converges to the empirical test risk as the labeled subset grows to the full test set.Source: §2.1 · p. 2

2.2. Actively Sampling Test Points

• Importance sampling corrects active-selection bias and, with a suitable proposal, can reduce risk-estimate variance relative to uniform sampling and uncorrected active sampling.The proposal may depend on previously acquired test data, the test set, training data, and the trained model.Source: §2.2 · p. 3

3.1. General Framework

• The ideal oracle proposal samples in proportion to each point’s true loss, yielding a single-sample zero-variance risk estimate, but is intractable.The true output distribution is unavailable before labeling, so the proposal must be approximated for unlabeled points.Source: §3.1 · p. 3
• Surrogates approximate the intractable proposal by identifying high-loss regions through uncertainty, diverse predictions, and information from newly labeled test points.The framework permits generative or discriminative surrogates, with uncertainty represented explicitly or implicitly.Source: §3.1 · p. 3

3.2. Illustrative Example

• In the illustration, points are sampled in proportion to approximate expected loss; the estimator remains unbiased, and closer loss estimates reduce its variance.The surrogate is retrained as new labels arrive.Source: §3.2 · p. 3, Figure 2 · p. 4

3.3. Deriving Acquisition Functions

• For regression, the acquisition uses the squared discrepancy between model and surrogate means plus predictive variance, which includes epistemic and aleatoric uncertainty.These quantities can be approximated without an explicit surrogate parameter distribution.Source: §3.3 · p. 4
• For classification, the section derives strategies for cross-entropy and accuracy, with the cross-entropy acquisition given by the surrogate’s marginal predictive distribution against the evaluated model.Source: §3.3 · p. 4

3.4. Tactics for Obtaining Good Surrogates

• Effective surrogates should represent both uncertainty types, use available data, and diversify predictions; retraining can add information and produce a more diverse ensemble.The benefits of retraining are especially large with very little data, while ensembling can help with more data.Source: §3.4 · pp. 4–5
• Using the original model as its own surrogate saves acquisition compute but is generally not recommended; it can work surprisingly well when its uncertainties reflect true loss.More complex strategies still outperform it, and a fixed surrogate distinct from the evaluated model typically provides noticeable benefits.Source: §3.4 · p. 5

• Adaptive experimental design selects informative data, and active learning applies this idea to choosing which training points receive labels (Houlsby et al., 2011; Sener & Savarese, 2018).Source: §4 · pp. 5–6
• Test-data acquisition has largely been neglected; Nguyen et al. (2018) considered vetting noisy labels but did not acknowledge substantial bias, while Farquhar et al. (2021) provide an unbiased estimator without active-testing acquisition strategies.Source: §4 · p. 6
• Classifier evaluation also uses stratified sampling, while prior importance-sampling work relied on replacement and did not consider surrogates for more effective acquisition.Active testing could improve stratified methods by selecting labels actively within strata.Source: §4 · p. 6

5.1. Synthetic Data

• After only 5 acquired labels, active testing’s standard deviation matched i.i.d. acquisition at step 40, and its estimates were unbiased.Both estimators converge to the full-test-set empirical loss when the entire test set is acquired; the variance difference matters at smaller acquisition counts.Source: §5.1 · pp. 6–7, Figure 3 · p. 6

5.2. Surrogate Choice Case Study: Image Classification

• A random-forest surrogate outperformed all strategies for ResNet-18 on Fashion-MNIST, despite being cheaper and not necessarily more accurate.For Radial BNN on MNIST, it beat i.i.d. acquisition but did not improve on predictive entropy.Source: §5.2 · p. 7

5.3. Large-Scale Image Classification

• Ensembles avoid most high-loss, overconfident errors that predictive entropy misses, improving the loss approximation used by the weighted estimator.The figure illustrates this for ResNet-18 on CIFAR-100.Source: §5.3 · p. 8, Figure 5 · p. 8
• About 4× efficiency gains on less complex datasets and nearly 2× on CIFAR-100; active testing also improves accuracy estimation.Across all cases, active testing has lower median squared error than i.i.d. sampling.Source: §5.3 · p. 8, Figure 6 · p. 8

5.4. Diversity and Fidelity in Ensemble Surrogates

• Both surrogate fidelity and diversity improve performance, but diversity has the larger observed effect; five ResNet-18s outperform one ResNet-50.The best performance uses the most diverse and complex surrogate.Source: §5.4 · p. 8, Figure 7 · p. 9

5.5. Optimal Proposals and Unbiasedness

• Sampling proportional to true loss with the LURE estimator gives exact single-sample estimates, while omitting bias correction yields biased, overestimated risks.The true-loss proposal requires an oracle and is not practicable.Source: §5.5 · pp. 8–9, Figure 8 · p. 9

5.6. Active Testing vs. Active Learning

• Predictive entropy outperformed mutual information for active testing because mutual information targets reducible uncertainty, not all uncertainty relevant to evaluation.This contrasts active testing’s acquisition needs with those of active learning.Source: §5.6 · p. 9

5.7. Practical Advice

• Deep-learning ensemble surrogates appear robustly sample-efficient; avoid relying only on the original model unless its uncertainties are trustworthy.Retrain when labeling is costly relative to retraining; otherwise, not retraining can be an option, especially with few acquired labels relative to training data.Source: §5.7 · p. 9

6. Conclusions

• Active testing provides principled acquisition functions for estimating test loss and accuracy with fewer labels.The authors describe the underlying idea as an initial contribution with substantial scope for further development and alternative approaches.Source: §6 · p. 9

A.1. Synthetic Data

• Active testing improves sample-efficiency in two additional synthetic regression settings: sinusoidal data with non-uniform x densities and quadratic data.These experiments use a Gaussian process on both datasets.Source: §A.1 · p. 12

A.2. Radial BNN on MNIST

• With 50,000 training points, predictive-entropy active testing yields large sample-efficiency gains over i.i.d. acquisition for a well-calibrated Radial BNN on MNIST.The authors use predictive entropy because they observe it to be well-calibrated in this setting.Source: §A.2 · p. 12

A.3. Uncertainties and Statistical Significance

• At the last displayed acquisition step, signed-rank tests reject the null at the 5 × 10^-3 confidence level for every comparison reported.The tests compare the best active-testing strategy with the other shown methods.Source: §A.3 · p. 12

B.1. Further Details on Clipping

• Proposal probabilities are floored at α = 0.2 times uniform-acquisition probability to keep every remaining point’s probability nonzero and preserve LURE unbiasedness.The floor prevents overly small acquisition values.Source: §B.1 · p. 12

B.2. Computational Complexity

• Acquisition-function evaluation costs O(MN) across M acquisitions because each step scores the remaining test-pool points.The number of remaining points decreases from N across successive iterations.Source: §B.2 · pp. 12–13
• Retraining after every acquisition costs O(M|Dtrain| + M^2) across M steps, under the worst-case schedule K = M.This schedule uses each newly acquired label to update surrogate predictions.Source: §B.2 · p. 13
• The total computational complexity is O(M|Dtrain| + MN) for M acquisitions, training-set size |Dtrain|, and test-pool size N.This simplifies the bound O(M|Dtrain| + M^2 + MN), using M ≤ N.Source: §B.2 · p. 13

C. Details on Experiments

• Neural-network experiments use PyTorch and a mix of RTX 2080, Titan RTX, and K80 GPUs, with no experiment requiring multiple GPUs.Synthetic-data experiments can run on CPUs using conventional hardware.Source: §C.1 · p. 13
• Wilcoxon signed-rank tests compare the squared differences of the best method against each other shown approach.The alternative hypothesis is that the best method has lower squared error.Source: Table 1 · p. 14
• The supplied Fashion-MNIST entry pairs Predictive Entropy and Mutual Information with 400 and 8.779 × 10−4, respectively.The passage does not clarify what the two values measure.Source: §C.1 · p. 14

C.2. Figures 1 to 3: Synthetic Data

• Synthetic regression uses Gaussian-process, quadratic, and sinusoidal data, while two-moons classification uses a random forest; the experiments repeatedly redraw train/test splits.Regression uses 5 training and 45 test points across 5000 experiments; two-moons uses 50 training and 450 test points across 2500 experiments.Source: §C.2 · p. 14

C.3. Figure 4: Radial BNN/ResNet-18 on MNIST/Fashion-MNIST

• Figure 4 experiments sample 250 training and 5000 test points, acquire 1000 test labels, and compute full test loss over all 5000 points.The appendix reports 1085 Radial BNN runs on MNIST and 872 ResNet-18 runs on Fashion-MNIST.Source: §C.3 · pp. 14–15

C.4. Figure 5: ResNet-18 on CIFAR-100

• Figure 5 uses the CIFAR-100 ResNet-18 setup described for Figure 6, with reported model accuracy of 72 to 76 percent.The reported setup takes about 12 minutes and includes 692 runs.Source: §C.4 · p. 16

C.5. Figure 6: ResNet-18/WideResNet on Fashion-MNIST, CIFAR-10/Cifar-100

• Figure 6 evaluates ResNet-18 and WideResNet models using repeated test-set samples; increasing ensemble size sometimes improved results.The appendix reports 93 to 94 percent accuracy for ResNet-18 on Fashion-MNIST, 92 to 93 percent on CIFAR-10, and 78 to 80 percent for WideResNet on CIFAR-100.Source: §C.5 · pp. 16–17

C.6. Figure 7

• Figure 7 uses the CIFAR-10 Figure 6 setup with a ResNet-18 main model and “train ensemble”-style surrogates.Source: §C.6 · p. 17

C.7. Figure 8; C.8. Figure A2: Radial BNN on MNIST.

• Figures 8 and A.2 use Radial BNNs with larger data setups; reported validation accuracy is about 88 to 92 percent on Fashion-MNIST and about 98 percent on MNIST.The appendix reports 962 Fashion-MNIST runs and 992 MNIST runs.Source: §C.7 · p. 17, §C.8 · p. 17

D. Comparison to Active Risk Estimation by Sawade et al.

• ARE and active testing both acquire test-pool labels, but ARE uses the original model rather than surrogate models and does not fully accommodate the pool-based setting.The appendix says active testing outperforms ARE in practice.Source: D. Comparison to Active Risk… · p. 17, §D.1 · pp. 17–18

D.1. Background

• ARE’s importance-sampling estimator requires sampling with replacement for unbiased estimates, so test points may be sampled repeatedly.Source: §D.1 · p. 18

Optimal Behavior

• With oracle risk and true outcome distributions, ARE improves over i.i.d. acquisition but lacks active testing’s single-sample, zero-variance behavior.Source: §D.2 · p. 19

Performance in Practice

• Active testing clearly outperforms ARE in practice, even though larger test sets may shrink the gap.The figure compares means and standard deviations across runs; the authors attribute the expected gap reduction to replacement sampling mattering less with larger test sets.Source: §D.2 · p. 19

Redefining Acquisitions Steps

• Counting only novel ARE acquisitions slightly improves its displayed performance, but it still performs far worse and requires significantly more queries.For Figure D.1(c), ARE takes (9.8 ± 3.1) times as many queries as test-pool samples.Source: §D.2 · p. 20