Source-linked AI summary
FaultLens: Learning Compact Behavioral Test Suites for Generated Operational Programs
Zeming Liu, Hang Lyu, Jingtao Zhang
TL;DR
Generated operational programs require validation beyond a few examples, yet exhaustive suites can be costly. FaultLens learns compact probe orderings from executed historical evidence, combining fault-driven greedy selection with structural diversity. Across future programs and withheld fault families, compact suites retain high fault coverage while remaining an auditable evidence mechanism rather than a correctness proof.
Problem
Generated operational programs can pass parsers and a few examples while exhaustive behavioral validation becomes expensive across boundaries, interactions, and external systems.
Method
FaultLens executes a rich finite probe domain, stores sparse fault–probe outcomes, and learns alternating active–diversity orderings from earlier generated programs.
Results
32 probes retain 576/582 (99.0%) dynamically killable fault coverage across future programs, while diversity raises whole-fault-family macro coverage from 84.6% to 94.9%.
Takeaways & Limitations
FaultLens provides prioritized, auditable behavioral evidence with explicit budgets, evidence sources, generalization splits, and misses.
Takeaways & Limitations
Controlled transformations may not represent real-world fault prevalence, and finite-domain equivalence does not imply equivalence for every possible input.
Abstract
from arXiv · showhide
Generated operational programs are often validated with either a few hand-written examples or exhaustive regression suites. The former can miss sparse boundary and interaction faults, while the latter can be unnecessarily expensive. We introduce FaultLens, a method for learning compact behavioral test suites while preserving an auditable connection to executed evidence. It executes a rich probe domain once, stores the fault-probe kill relation as a sparse outcome cache, and learns probe orderings only from earlier program generations. A fault-driven greedy component exploits known kill structure, while a mutation-independent diversity component covers probe families, cases, templates, and temporal bins. Their alternating hybrid remains useful when a new program contains a fault mechanism absent from ordering construction. We evaluate twenty generated operational policies across four environments, ten execution seeds, 1,200 measured run summaries, 2,160 controlled program transformations, and 4,120,200 executed program-probe pairs. Of 1,960 intended faulty transformations, 1,779 alter a contract or output somewhere in the finite audit domain; 200 additional controls preserve behavior. A 32-probe hybrid learned on generations 1-3 covers 576/582 (99.0%) dynamically killable faults in generations 4-5 using 1.2-2.0% of the exhaustive domain. With an entire fault family withheld from training, diversity raises scenario-family macro coverage from 84.6% to 94.9%. In a downstream deployment study, a conservative admission rule reduces severe tail regressions from 15/20 program-environment groups to 0/20. FaultLens provides a prioritized evidence mechanism, not a proof of correctness, and makes its budget, evidence source, generalization split, and misses explicit.
1 Introduction
FaultLens targets the gap between weak hand-written examples and costly exhaustive validation by learning compact probe orderings from executed historical evidence. It combines fault-driven exploitation with structural diversity to improve transfer to future programs and unseen fault mechanisms.
- Generated operational programs can satisfy syntax and legal-action checks while containing shifted boundaries, order dependencies, stale state, damaged thresholds, or unsafe resource selections.
- The paper asks whether historical generated programs can support a much smaller probe suite that retains fault sensitivity on future programs.
- FaultLens greedily covers training mutants using executed kill relations while a second ranking covers structural features independently of mutant outcomes.
- The hybrid alternates unique probes from both rankings, splitting its nominal budget between evidence exploitation and structural exploration.
- The evaluation uses cross-program and whole-fault-family separation protocols to test generalization and selector overfitting.
- 32 probes cover 576/582 (99.0%) dynamically killable held-out faults, while structural diversity improves generalization to unseen fault mechanisms.
2 Problem Setting
FaultLens validates generated operational programs over a finite, interpretable probe domain against behavioral contracts and controlled transformations. The study asks which probes expose faults, how coverage changes with budget, whether diversity transfers to withheld fault families, and whether compact validation supports safer deployment decisions.
- At each decision round, a program maps observations to an action set under a resource budget.
- Domain contracts require deterministic outputs, duplicate invariance for set-like observations, and permutation invariance when order is semantically irrelevant.
- Each probe records its round, observation multiset, family, and human-readable case, while validation checks action type, uniqueness, budget, and catalog membership.
- A probe kills a controlled mutant when the normalized output differs from the unmodified program or becomes invalid.
- The selector chooses an ordered probe prefix under budget using only permitted training programs and fault families.
- The study evaluates probe-family exposure, shrinking-budget cross-program coverage, whole-family generalization, and downstream admission safety.
3 Method
FaultLens constructs rankings from executed historical evidence, executes selected probes on each new program, and uses a sparse cache for exact subset evaluation relative to the frozen full-suite trace. Fail-closed checks precede prioritization, while diversity offsets active selection’s tendency to over-specialize.
- Workflow: Historical programs construct executed outcome evidence and a probe ranking; new programs execute the selected probes rather than using cached outputs.
- Contracts Before Prioritization: The implementation performs static parsing, entry-point and mutation checks, fresh-process execution, resource limits, and repeated output validation.
- Contracts Before Prioritization: Prioritization operates only on baseline programs passing static and metamorphic contracts, and static-only faults do not consume dynamic probe budget.
- Probe Domain: The exhaustive domain combines rounds with threshold neighborhoods, duplicate and triplicate observations, template permutations, mixed pairs, and unknown templates.
- Sparse Executed Outcome Cache: The cache stores static outcomes and killing-probe IDs per mutant, enabling counterfactual subset evaluations without rerunning programs or systems.
- Sparse Executed Outcome Cache: The cache is exact relative to the frozen full-suite trace but does not establish equivalence under arbitrary probe reorderings for hidden-state programs.
- Rankings: Active greedy selection covers newly killed mutants and can over-specialize, whereas diversity covers probe families, cases, templates, temporal bins, and interactions.
4 Study Design
The study evaluates generated resource-selection programs across four workload environments using controlled transformations, repeated execution seeds, and complete behavioral domains. It separates cross-program and whole-fault-family evaluation while pairing behavioral validation with downstream operational admission criteria.
- Case Study: 20 program–environment groups span four workload environments, each using a 120k-row SQLite instance over 60 decision rounds.The concrete action is database-index activation, while the validator observes rounds, records, catalogs, and budgets.
- Case Study: 1,200 frozen measurement rows come from five generation runs, ten execution seeds, and six operational baselines.Generated artifacts are treated as fixed subjects rather than evaluated for generation quality.
- Transformation Corpus: 2,160 controlled transformations comprise 98 intended faults per program and ten behavior-preserving controls, varying temporal, structural, threshold, and action parameters.Faults span thirteen families, while controls span five families.
- Transformation Corpus: 4,120,200 program–probe pairs yield 1,779 effective fault instances from 1,960 candidates, while all 200 controls preserve static contracts and outputs.181 fault candidates are equivalent on the finite domain; dynamic and static detection channels are reported separately.
- Generalization Protocols: Cross-program evaluation trains on generations 1–3 and tests on generations 4–5, with 582 dynamically killable held-out instances.Whole-family holdout removes one fault family during ordering construction and reports macro coverage over 42 nonempty environment–fault-family cells.
- Downstream Operational Admission: Downstream admission requires behavioral-contract passage, improved calibration loss, and one-sided 95% bootstrap bounds satisfying fixed Safe or Balanced operating profiles.Safe permits at most 5% p95 harm; Balanced permits 30%, and both permit 0.02 normalized violation harm.
5 Results
Fault evidence is distributed across probe families, so compact rankings must balance observed fault kills with structural exploration. On held-out programs and fault families, hybrid suites retain high coverage at a small fraction of the exhaustive domain, while downstream admission reduces severe tail regressions.
- 5.1 RQ1: Evidence Is Distributed Across Probe Families: 1,662 effective faults do not change a static contract, and different fault types require multiplicity, threshold, multi-observation, temporal, or catalog coverage.No individual probe family dominates all fault rows, motivating a rich complete domain and explicit structural exploration.
- 5.2 RQ2: Compact Suites Transfer Across Programs: 571/582 (98.1%) held-out faults are covered by active and hybrid rankings at 16 probes, versus 89.2% for random.Active greedy is strongest at budgets 4 and 8 because early selections exploit observed kills.
- 5.2 RQ2: Compact Suites Transfer Across Programs: 576/582 (99.0%) held-out faults are covered by the 32-probe hybrid using 1.2% of the 2,700-probe domain or 2.0% of a 1,620-probe domain.Composite, Cycle, and Rare reach 100%; the six remaining misses occur in Burst.
- 5.2 RQ2: Compact Suites Transfer Across Programs: At 128 probes, active reaches 100% while hybrid remains at 99.1%, showing that hybrid exploration is not uniformly superior.Very small budgets may favor exploitation when the historical fault model is trusted; longer active prefixes or the full domain suit higher-assurance deployment.
- 5 Results: The study claims reduced executed probe count and validation payload, not a measured 50× wall-time speedup.The full batch audit has median wall time 59.1 ms per baseline program, with process startup dominating this micro-policy case study.
- 5.3 RQ3: Diversity Helps on Unseen Fault Families: 84.6% macro coverage for active rises to 94.9% for hybrid across 42 nonempty environment–family cells under whole-fault-family holdout.The largest gains occur for threshold corruption, round shifts, and partial schedules; diversity is equal or better on all eleven dynamic families.
- 5.4 RQ4: Downstream Admission Reduces Tail Regressions: The Safe profile achieves loss 0.0269 with zero observed severe regressions, while deploying every free program yields loss 0.0838 and regressions in 15 of 20 groups.Margins through 0.20 have zero observed severe regressions, whereas four appear at 0.30.
6 Analysis and Ablations
Ablations show that nonredundant set cover and structural diversity are important for transfer beyond the historical fault distribution. The conclusions remain bounded by the chosen structural features and by layered validation requirements.
- 6.1 Why Frequency Is Not Enough: 81.8% coverage from frequency prioritization versus 98.1% for active set cover at budget 32 quantifies the value of updating the uncovered set.Frequency prioritization counts individual training kills but does not account for redundant coverage.
- 6.2 Why Diversity Helps: The whole-family holdout reveals a 10.3-point macro gap, while ordinary cross-program rankings differ by less than one point at budget 32.Diversity protects against changed relationships between fault mechanisms and informative probe structure, rather than primarily new source syntax.
- 6.2 Why Diversity Helps: The conclusion is limited to five-round bins, probe cases, and template tokens; other programs may require topology, permission, API, or concurrency features.The selected structural features are natural for this case study but are not universal.
- 6.3 Layered Validation: Static contracts detect 117 effective transformations but miss 1,662 output-only faults, whereas compact probes provide behavioral evidence under a cost budget.Task measurements separately establish whether a valid program should be deployed at a declared risk operating point.
- 6.4 Evidence and Reproducibility: The 27 MiB sparse cache supports recomputing budget curves and holdouts without rerunning programs or database execution, while retaining concrete killing probe IDs.This preserves an auditable connection between aggregate results and executed evidence.
7 Limitations
FaultLens’s evidence is bounded by its program domain, mutation design, execution assumptions, operational sampling, and security model. These boundaries limit what its compact suites and downstream decisions can establish.
- Program domain: Twenty short deterministic Python policies across four environments define the study’s scope.Tool use, schema manipulation, distributed coordination, and concurrency require stronger isolation and different structural features.
- Mutation representativeness: Controlled transformations measure fault sensitivity, not real-world fault prevalence.The thirteen fault families may favor the motivating grammar; holdouts reduce leakage but do not replace real incident corpora.
- Finite-domain equivalence: Finite-domain equivalence does not establish equivalence for every possible input.The 181 excluded candidates are equivalent only on the complete audit domain, and the 200 controls do not cover all semantics-preserving transformations.
- State and ordering: Cached subset coverage is exact for frozen full-domain executions but may differ under another execution order for hidden-state programs.The method rejects explicit state and tests repeatability, but this is not a formal purity proof.
- Operational statistics: Bootstrap bounds cannot correct deployment distribution shift, and the 5% severe-regression threshold is an operational definition rather than a theorem.Stronger distributional guarantees require additional assumptions.
- Security: AST restrictions and resource limits reduce accidental damage but do not safely execute adversarial code.Untrusted deployment requires OS isolation, credential separation, syscall controls, and security review.
8 Related Work
FaultLens builds on property, metamorphic, behavioral, mutation, regression-prioritization, and operational-risk work while focusing on auditable ordering over finite scenario-derived probes. Its distinctive evaluation tests transfer across programs and withheld fault families.
- Testing foundations: Property, metamorphic, and behavioral testing provide precedents for generated tests, relations, and organized behavioral capabilities.FaultLens instead learns an ordering over probes from a finite, scenario-derived grammar.
- Mutation testing: Mutation analysis supplies controlled-fault evaluation, while FaultLens records a sparse kill relation and tests transfer beyond prioritization transformations.Program and operator holdouts assess whether compact subsets generalize.
- Regression prioritization: The active component is a deterministic set-cover heuristic, while diversity reserves budget for structural exploration under correlated mutation evidence.This combines early-fault prioritization with broader probe coverage.
- Operational learning and risk: Operational-risk work motivates fallback mechanisms and tail-focused evaluation, but FaultLens separates compact behavioral evidence from the downstream performance rule.Bootstrap intervals quantify sampling uncertainty rather than distribution-free deployment guarantees.
9 Reproducibility
The reproducibility artifact separates executed evidence from replayed selection experiments. It records the inputs needed to audit both the full run and cached reconstructions without rerunning programs.
- Artifact: The artifact records stable probe IDs, transformation parameters, outcomes, hashes, seeds, bootstrap labels, and every plotted CSV row.The complete source run executes 4,120,200 program–probe pairs.
- Reproduction targets: A full run reconstructs executed outcomes from frozen program sources, whereas a replay run reconstructs selection experiments from immutable evidence.This separation supports auditing prioritization logic without trusting a synthetic performance model.
10 Conclusion
FaultLens learns compact, auditable behavioral evidence for generated operational programs rather than proving correctness. Its value lies in explicit budgets, evidence sources, generalization splits, and deployment operating points.
- Conclusion: 32 probes retain 99.0% fault coverage across future programs while using 1.2-2.0% of the exhaustive domain.The remaining misses make the suite a prioritized evidence mechanism, not a correctness proof.
- Conclusion: With an entire fault family withheld from training, diversity improves scenario-family macro coverage from 84.6% to 94.9%.The result supports reserving budget for mutation-independent structural exploration.
A Per-Environment Budget Curves
Per-environment curves reveal substantial heterogeneity: the hybrid reaches full coverage by 16 probes in three environments, while Burst requires a longer prefix and contains all six misses at budget 32. Downstream decisions likewise vary across environments rather than reflecting a uniform reject-all policy.
- Per-environment coverage: 100% coverage is reached by 16 probes in Composite, Cycle, and Rare, whereas Burst requires a longer prefix.Burst accounts for all six cross-program misses at budget 32.
- Per-environment coverage: All six cross-program misses at budget 32 occur in Burst.The environment-specific curve exposes heterogeneity that a weighted aggregate could conceal.
- Downstream admission: The twenty downstream decisions include free generated programs and restricted references, with especially large calibration upper bounds and held-out tail ratios in Composite and Cycle.Several Burst programs instead improve tail latency.
- Downstream admission: The aggregate downstream result does not arise from a uniform reject-all policy.The plotted decisions include both adverse outcomes in some environments and tail-latency improvements in others.