\textsc{TestifAI}: Tomography-Based Testing for Deep Learning Systems

Arooj Arif; Tobias Hartung; Elena Botoeva; Alexandros Koliousis

Source-linked AI summary

\textsc{TestifAI}: Tomography-Based Testing for Deep Learning Systems

Arooj Arif, Tobias Hartung, Elena Botoeva, Alexandros Koliousis

arXiv:2608.18900v1Aug 19, 2026cs.AI

TL;DR

Testing robustness across combinations of input perturbations is difficult because exhaustive evaluation is combinatorial. TestifAI reconstructs higher-order robustness from lower-order tests, predicting 3- and 4-perturbation outcomes with less than 7% aggregate estimation error across five benchmarks.

Problem
Existing robustness testing does not systematically capture interactions among multiple perturbations and severity levels, despite models needing evaluation under intended operating conditions.
Method
TestifAI trains an auxiliary predictive model on all first- and second-order perturbation tests to estimate robustness across untested higher-order configurations.
Results
Less than 7% aggregate estimation error was achieved when predicting third- and fourth-order test outcomes from lower-order observations across five vision and language benchmarks.
Takeaways & Limitations
TestifAI supports principled extrapolation of robustness to unseen multi-perturbation scenarios instead of treating deep learning testing solely as a combinatorial coverage problem.
Takeaways & Limitations
TestifAI may misestimate robustness when higher-order effects are not predictable from single or pairwise tests, so practitioners may need a modest third-order evaluation budget.

Abstract

from arXiv · show

As AI systems are increasingly deployed in safety-critical application domains (e.g., autonomous driving), associated risks increase too. Deep learning models underlying modern AI systems, therefore, must undergo thorough testing to ensure their correct behaviour. A single robustness test involves thousands of inferences to empirically verify if a model's outputs remain stable under a bounded perturbation of its inputs. However, existing testing frameworks lack the means to systematically explore and summarise robustness across a combinatorial space of perturbations. We propose TestifAI, a deep learning testing framework for efficient and accurate estimation of robustness against combinations of perturbations. TestifAI enables users to specify operational conditions as structured spaces of semantic input perturbations (e.g., image blur, brightness and zoom) and discrete severity levels (e.g., low, medium and high). Users can query model robustness for any combination (e.g., "low blur, high brightness, and medium zoom"). To achieve efficiency and accuracy, TestifAI introduces partial model tomography, a novel approach to reconstructing model behaviour in a multi-perturbation space from tests that apply only a small number of perturbations (lower-order projections). To estimate robustness against at least three perturbations, TestifAI trains an auxiliary model on the results of tests involving up to two perturbations only, avoiding execution of an exponential number of tests. Our experiments on five image and language classification tasks show that TestifAI can predict higher-order (3 and 4 perturbations) test outcomes from low-order (1 and 2 perturbations) observations with an aggregate robustness estimation error of less than 7%, while reducing the number of inferences by 60-80%.

1 Introduction

TestifAI addresses the challenge of testing deep learning robustness across interacting perturbations by learning higher-order behaviour from lower-order tests. Evaluated on five vision and language classification tasks, it estimates robustness across 3-way and 4-way perturbation combinations with minimal approximation error.

Deep learning models are increasingly deployed in high-stakes applications, including autonomous vehicles, AI chatbots, and other real-world systems.
Input perturbations can cause deep learning models to fail, including lighting, geometric, weather, spelling, character, and paraphrase changes.
Thorough testing must assess model behaviour under intended operating conditions, including assumptions about data distributions and architectural inductive biases.
Existing testing techniques overlook interaction effects or lack mechanisms to explore them, while static benchmarks treat perturbations as atomic units.
TestifAI uses partial model tomography to predict higher-order robustness from tests involving only one or two perturbations at a time.
Five classification benchmarks—four vision and one language—show that a random forest trained on first- and second-order tests estimates robustness across 3-way and 4-way combinations with minimal approximation error.

2 Testing Deep Learning Models

Testing deep learning models evaluates whether predictions remain correct under controlled, label-preserving perturbations that shift inputs and probe learned decision boundaries. The central challenge is exploring higher-order perturbation combinations despite exponentially many possible tests.

Testing Deep Learning Models: Testing applies controlled, label-preserving input changes and checks whether a classifier maintains consistent predictions for the same example.Perturbations can reveal brittleness when they alter the model’s approximation of P(Y|X).
Testing Deep Learning Models: A perturbation configuration specifies severity levels for parameterised perturbations, whose composition defines the transformed input tested by the model.Configurations can combine multiple perturbation functions, such as image rotation, at specified severity levels.
Testing Deep Learning Models: The robustness score is the fraction of test-set inputs on which the model remains correct under a given perturbation configuration.Aggregate robustness averages robustness scores across a test suite.
Testing Deep Learning Models: Real-world inputs vary across multiple axes, while manually anticipating all composite perturbations is infeasible.Examples include combined weather effects, co-occurring text transformations, and image corruptions used during training.
Testing Deep Learning Models: The number of possible combination tests grows exponentially with the number of perturbations, making exhaustive testing infeasible.Combinatorial Interaction Testing reduces this burden through covering arrays with t-way interaction coverage, but the remaining challenge is predicting higher-order behaviour.

3 Partial Model Tomography

Partial model tomography approximates higher-order robustness from lower-order tests in structured perturbation spaces, avoiding exhaustive combinatorial testing. It combines independence checks with predictive models to reduce testing cost while accounting for dependencies.

Tomography assumptions: TestifAI models atomic semantic perturbations with discrete severity levels and supports applying multiple perturbations in combination.These assumptions define the structured perturbation spaces used for partial model tomography.
Combinatorial bottleneck: 216 unordered and 916 ordered tests arise from three perturbations with six severity levels each, illustrating full tomography’s combinatorial cost.The ordered count grows when perturbation application order matters.
Independence-based estimation: Independence allows higher-order robustness to be estimated by multiplying lower-order robustness values, such as 0.9 × 0.8 = 0.72.The method tests this assumption empirically by comparing first-order predictions with observed second-order results.
Test reduction: 125 third-order tests, 58% of the total, can be estimated rather than executed when all first-order perturbations are independent.Partial independence can similarly support direct estimation for higher-order tests whose constituents are pairwise independent.
Dependency-aware prediction: Random forests trained on first- and second-order results predict third-order outcomes and beyond while accounting for dependencies.This general predictive approach reduces tomography’s computational cost in user-specified test environments.

4 TestifAI

TestifAI performs partial tomography by executing first- and second-order perturbation tests, training an auxiliary model from their outcomes, and predicting robustness across higher-order configurations. Users define perturbation environments and query estimated robustness over selected regions of the resulting tomography space.

Framework workflow: TestifAI’s four-stage workflow lets users select perturbations and severities, execute first- and second-order tests, train a random-forest predictor, and query the estimated tomography space.The predictor approximates the model’s full tomography space from lower-order test results.
User interaction: Users can customise perturbations and severity levels, then express robustness queries as Boolean combinations of constraints such as zoom > 2 or brightness = 5 ∧ blur = 1.Queries select subsets of the tomography space, whose aggregate robustness TestifAI computes from measured or predicted test robustness.
Higher-order prediction: TestifAI predicts robustness for higher-order tests Θ⩾3 using empirical observations from lower-order tests Θ⩽2.The predictive model maps each perturbation configuration to a predicted robustness score.
Partial tomography: O(k^2m^2) tests replace the exponential O(m^k) cost of full tomography when k perturbations each have m severity levels.TestifAI evaluates all configurations with at most two active perturbations before prediction.
Extensions: An analogous surrogate model can estimate perturbation validity metrics from low-order observations, enabling users to exclude low-quality regions of Θ.The passage gives KID and BERTScore as example validity measures for perturbed images and text.

5 Evaluation · 5.1 Experimental setup

The evaluation examines TestifAI’s prediction accuracy, partial-tomography effectiveness, and sample efficiency across five classification benchmarks. Experiments compare partial-tomography predictions with exhaustive full-model tomography as ground truth.

5 Evaluation: Three research questions assess prediction accuracy, partial-tomography effectiveness, and sample efficiency.
5.1 Experimental setup: TestifAI uses Python 3.9, scikit-learn random forests, and sympy for Boolean query parsing and evaluation.Experiments ran on an MGHPCC GPU cluster with an NVIDIA Tesla T4 and CUDA 12.3.
5.1 Experimental setup: The evaluation covers five benchmarks comprising four vision classification tasks and one language classification task.
5.1.1 Benchmarks.: Each benchmark uses a publicly available pretrained classification model and associated dataset, documenting dataset size, perturbations, model, and classification accuracy.
5.1.1 Benchmarks.: The benchmarks test robustness to domain-specific perturbations, including image corruptions for digit, image, driving, and traffic-sign recognition, plus language-preserving variations for question answering.The evaluated models include LeNet-5, WideResNet-28-10, YOLOv11s, CNN-SE, and RoBERTabase.
5.1.2 Ground Truth.: Full model tomography is the baseline, exhaustively applying every perturbation configuration to every evaluation input to compute exact robustness scores.Figure 2 presents the resulting ground-truth robustness-score distributions.
5.1.2 Ground Truth.: Partial tomography estimates higher-order robustness from statistical patterns in the Θ1 and Θ2 distributions rather than exhaustively testing the full perturbation space.

5.2 Robustness estimation errors

TestifAI evaluates robustness estimation by comparing predicted and ground-truth scores, with positive errors indicating pessimism and negative errors indicating optimism. Across 216 three-perturbation tests, errors generally cluster near zero, while predefined aggregate queries assess robustness over progressively varied regions of the test space.

Error definition: Robustness estimation error is defined as predicted score minus ground-truth score; positive values indicate underestimated robustness, while negative values indicate overestimated robustness.The comparison is between predicted scores ˆr𝝈 and ground-truth values r𝝈.
Individual-test accuracy: Across 216 3D-tomography tests, most robustness estimation errors cluster tightly around zero across all five benchmarks.The box-and-whiskers plot summarizes each benchmark’s median, interquartile range, and full error spread.
Individual-test accuracy: For ì and 6, all prediction errors fall within [−0.05, +0.05].This result is reported for the 3D-tomography evaluation.
Aggregate robustness queries: Queries Q1–Q8 partition the test space into mutually exclusive regions, each covering exactly 12.5% of Θ, or 27 individual tests.These exclusive queries isolate specific regions and failure modes across perturbation severities.
Aggregate robustness queries: Queries Q9–Q12 provide inclusive regions ranging from 3.7% of the space, or 8 tests, to 100%, or all 216 tests.The progressively larger regions support fine-to-coarse analysis of robustness across the perturbation space.
Aggregate robustness queries: The aggregate-query evaluation tests whether a random forest trained only on low-order tests can estimate robustness degradation over increasingly complex subspaces without high-order test data.The predefined queries cover distinct portions of the perturbation space.

5.3 Effect of partial model tomography

Partial model tomography outperforms uniform random test selection for estimating robustness across 3D and 4D perturbation spaces. Ordered lower-order tests achieve accurate generalization with substantially less training data, supporting their value as sample-efficient predictors of higher-order behaviour.

Experimental design: Ordered sampling selects tests by increasing perturbation order, whereas random sampling selects the same number uniformly from the full perturbation space.Training-set sizes range from 10% to 100% of available tests in 10% increments.
Evaluation: MSE is computed on the remaining tests after applying each selected perturbation to every dataset input and recording model success or failure.The resulting robustness predictions are evaluated using mean squared error over predicted robustness scores.
Results: Ordered sampling consistently achieves lower MSE than random sampling, with robust generalization typically reached using only 20–30% of the perturbation space.Random sampling converges more slowly and has greater variance, especially at small sample sizes; third-order data further improves MSE.
Interpretation: The results support partial model tomography as a principled, sample-efficient alternative because lower-order projections contain informative structure about the full perturbation space.Stable and predictable responses can provide signals that generalize well to higher-dimensional settings.

5.4 Partial tomography efficiency

Partial tomography reconstructs 3D robustness spaces with substantially fewer inferences than exhaustive full tomography. Without early stopping, it reduces inference counts by at least 58%, while early stopping lowers costs further across most benchmarks.

Full tomography achieves zero robustness estimation error but requires |Θ| × |D| inferences, including 2,160,000 inferences in one example.It exhaustively evaluates all tests in Θ.
Partial model tomography evaluates at most |Θ⩽2| × |D| inferences, reducing total inference counts by at least 58%.The evaluation focuses on 3D tomography and uses total inferences as a proxy for runtime.
58%: Without early stopping, partial tomography reduces total inferences by 58% for every benchmark.This compares partial tomography against full 3D tomography.
72.7%, 61.6%, 73.15%, and 83%: Early stopping lowers inference costs to these levels for four benchmarks, with no additional improvement observed for the fifth.The reported benchmark-specific costs are 72.7% for ô., 61.6% for ì, 73.15% for , and 83% for -; computing full 3D tomography for 6 required approximately 47.3 hours.

6 On Validity & Quality Estimation

TestifAI assumes domain experts define deployment-relevant perturbations and severity levels, rather than assessing the semantic validity of the perturbation space itself. It can nevertheless estimate input-quality effects from low-order tests and use quality constraints to distinguish plausible from implausible robustness failures.

Validity assumptions: TestifAI does not assess whether higher-order perturbations are semantically valid, relying on domain experts to specify deployment-relevant perturbations and severity levels.This follows the practice of existing robustness benchmarks.
Quality metrics: KID for vision and BERTScore for language quantify deviation from unperturbed inputs as proxies for perturbation validity.The authors found these metrics integrated best with partial tomography and early stopping compared with other evaluated quality metrics.
Quality estimation: Partial tomography estimates higher-order quality scores by training a random-forest regressor on tests involving at most two perturbations.The regressor uses the same feature representation as robustness prediction and replaces robustness scores with quality scores.
Quality estimation: Mean absolute 4D quality-estimation error is approximately 0.1 for some benchmarks and near zero for others, indicating that validity is structured and predictable from low-order observations.The reported variation is measured across benchmarks.
Validity-aware interpretation: Equal quality scores can correspond to both plausible and invalid inputs, so perturbation validity depends on the specific combination rather than quality magnitude alone.Examples also show that compounding different perturbations can differ from applying one perturbation at higher severity.
Validity-aware querying: Users can refine queries with a predicted-quality threshold, excluding tests outside the desired validity range and distinguishing failures under plausible conditions from those caused by implausible inputs.Choosing thresholds requires domain expertise, and high severity sums do not necessarily imply invalidity because multiple perturbations may co-occur in practice.

7 Discussion

TestifAI is applicable when semantic perturbations can be discretized and robustness interactions are sufficiently captured by single and pairwise tests. Its partial-tomography principles also extend beyond classification when system-level metamorphic tests and quality-based oracle signals are available.

Applicability: TestifAI requires a user-defined, semantically meaningful perturbation space with discrete severity levels; coarse binning can capture trends, but gradient-based attacks fall outside its scope.The framework is best suited to perturbations that support queryable partial tomography.
Limitations: TestifAI may misestimate robustness when higher-order effects are not predictable from single or pairwise tests, but a modest Θ3 evaluation budget can improve predictions.Automatically identifying cases with higher-order coupling remains future work.
Generality: Partial tomography extends across model types and modalities beyond classification when perturbations are discretizable and system-level metamorphic tests provide acceptable-behaviour signals.Regression and generation examples use KID and BERTScore as quality-based oracle signals for perturbed images and text.

8 Related Work

TestifAI differs from prior deep-learning testing methods by systematically reasoning over structured combinations of semantic perturbations and estimating higher-order robustness from second-order tests. It complements existing perturbation benchmarks and augmentation resources while remaining distinct from neuron-coverage, input-prioritization, combinatorial-covering, and formal-verification approaches.

Related testing methods: Prior testing methods do not support systematic reasoning over structured multi-perturbation spaces or querying robustness under diverse, interacting perturbations.These directions include training input generation, coverage-guided testing, test prioritisation, and combinatorial testing.
Related resources: Robustness benchmarks and linguistic perturbation resources provide complementary curated transformations that users can combine for systematic testing.Examples include MNIST-C, CIFAR-10-C, ImageNet-C, KITTI-C, nuScenes-C, Waymo-C, CURE-TSR, CheckList, and TextFlint.
Data augmentation: TestifAI systematically estimates aggregate robustness under structured multi-perturbation test environments, unlike stochastic augmentation methods that combine transformations during training.CutMix, AugMix, and PixMix expose models to simultaneously transformed inputs to improve robustness.
Coverage and prioritization: TestifAI prioritises system-level metamorphic tests consisting of structured semantic perturbation combinations, whereas neuron-coverage methods target activation coverage and input-prioritisation methods target likely misclassifications.Input-level prioritization can complement TestifAI’s early stopping strategy when users seek worst-case robustness.
Combinatorial interaction testing: TestifAI uses second-order tests to train a predictive model for multi-perturbation robustness rather than constructing high-dimensional covering arrays.It extends pairwise perturbation testing by approximating higher-order behaviour through 3D and 4D tomography.
Formal guarantees: TestifAI reports empirical aggregate robustness over a semantic region, unlike neural-network verifiers that seek certificates for all bounded perturbations of individual inputs.Its system-level metamorphic tests assess robustness under structured semantic perturbations rather than guaranteeing local robustness everywhere.

9 Conclusion

TestifAI reframes deep learning testing as predictive modelling of interactions among multiple perturbations. It uses lower-order test results to approximate robustness for higher-order perturbation combinations.

Conclusion: TestifAI treats deep learning testing as a predictive modelling task rather than a combinatorial coverage problem.The approach reasons about how multiple perturbations interact.
Conclusion: The surrogate model uses all combinations of up to two perturbations and their severity levels to estimate robustness for tests involving three or more perturbations.This lower-order training setup approximates higher-order test behaviour.
Conclusion: Experiments on five vision and language benchmarks evaluate TestifAI’s predictions for third- and fourth-order perturbation tests.The supplied passage introduces these benchmark evaluations but does not provide their quantitative outcomes.