PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Fixed Fault Models: Comparing LLM-Based and Rule-Based Fault Injection in OpenStack

Giuseppe De Rosa, Pietro Liguori, Domenico Cotroneo

arXiv:2609.08681v1cs.SE

TL;DR

Fixed fault catalogs are controlled but bounded by manually designed operators, leaving context-dependent defects outside existing templates unexplored. We compare two fine-tuned code LLMs with ProFIPy in OpenStack; on shared Nova targets, they show distinct failure profiles, with more Catastrophic LLM outcomes and more Silent ProFIPy outcomes.

  • Problem

    ProFIPy’s manually designed operators bound the explored fault space, so context-dependent defects that do not match an existing template may remain unexplored.

    Source: §I · p. 1

  • Method

    We compare Qwen2.5-Coder and DeepSeek-Coder with ProFIPy in OpenStack using the same testbed, workload, coverage measurements, and failure oracles.

    Source: §I · p. 1

  • Results

    Across the campaign, ProFIPy faults propagated more often among components; on shared Nova targets, LLM faults produced more Catastrophic failures, while ProFIPy produced more Silent failures.

    Source: §I · p. 2, in the failures they expose? · p. 4

  • Takeaways & Limitations

    LLM-based fault injection complements fixed-model SFI by reducing fault-model authoring effort and extending the observed failure space, without establishing generator-level superiority.

    Source: §VII · p. 8

    Context-specific mutation does not guarantee fault representativeness; adoption still requires scope control, execution evidence, representative workloads, failure oracles, and reproducible provenance.

    Source: §V · p. 6

Abstract

from arXiv · show

Software Fault Injection (SFI) supports testing of cloud systems by introducing software defects and observing their manifestation. Rule-based injectors such as ProFIPy provide controlled and reproducible source-level mutations but require fault patterns to be encoded manually. Large Language Models (LLMs) offer a data-driven alternative by generating context-dependent software faults. We compare two code LLMs, Qwen2.5-Coder and DeepSeek-Coder, with ProFIPy in OpenStack's Nova and Cinder services. On shared injection targets, activation and observable-failure rates are comparable, but operational profiles differ: LLM-generated faults produce more Catastrophic outcomes on Nova, whereas ProFIPy produces more Silent and Multi-component effects. The sampled LLM outputs also differ in how they manifest failure, while showing greater agreement in their propagation scope. These findings show that LLM-based fault injection extends the behavioral coverage of fixed fault models without establishing general superiority, and that practical adoption still requires controlled generation, runtime validation, system-level oracles, and reproducible experimental provenance.

I. INTRODUCTION

The paper compares generative and rule-based source-level fault injection to test whether LLM-generated mutations extend the operational behaviors exposed in OpenStack. On shared Nova targets, activation and observable-failure rates are similar, but failure profiles differ; overall, the approaches expose complementary behaviors.

I. INTRODUCTION

  • Cloud robustness assessment needs representative faults and complementary oracles because an apparently successful API response can leave a resource unusable or state inconsistent.A volume-attachment request may appear successful even when the volume is unusable; prior experiments detected such non-fail-stop failures through independent state assertions.Source: §I · p. 1
  • ProFIPy’s catalog provides valid, reproducible, controlled mutations, but context-dependent defects outside its manually designed operators may remain unexplored.Arbitrary syntactic mutations may not reflect developer defects and can lead to misleading conclusions [2].Source: §I · p. 1
  • LLMs generate context-dependent mutations from function context, but outputs may be invalid, equivalent, off-target, or incompatible with the target runtime and require explicit validation.For OpenStack Pike, the study compares Qwen2.5-Coder and DeepSeek-Coder with ProFIPy in Nova and Cinder, using a shared testbed, workload, coverage measures, and failure oracles.Source: §I · p. 1
  • Earlier LLM fault-generation work mainly assessed code-level properties, leaving unclear whether accepted faults expose operational behaviors beyond fixed catalogs or mainly cause immediate crashes.The study examines accepted faults under a common deployment, workload, and set of failure oracles; extension means additional behaviors observed in the evaluated campaign.Source: §I · p. 1
  • ProFIPy faults propagated among components more frequently, while LLM effects were predominantly local or caused immediate service disruption.Qwen and DeepSeek also differed in failure outcomes at the same locations, while target service and injection point more strongly shaped their system-level impact.Source: §I · p. 2

II. BACKGROUND & RELATED WORK

The background distinguishes source-level implementation-defect injection from runtime failure injection and mutation testing. Fault representativeness depends on how and where a transformation is activated and manifests.

A. Software Fault Injection

  • Source-level injection emulates persistent implementation defects, while runtime fault-injection approaches commonly model transient faults or failures such as exceptions, delays, failed calls, and message loss.Runtime approaches such as FATE and DESTINI, Molly, Filibuster, and Legolas complement source-level faults.Source: A. Software Fault Injection · p. 2
  • Mutation testing measures whether a test suite kills mutants, whereas dependability studies examine activation, detection, containment, reporting, and propagation.A useful testing mutant is not necessarily representative of a residual production fault.Source: A. Software Fault Injection · p. 2
  • Fault representativeness depends on the transformation, location, activation conditions, context, and failure behavior; catalogs provide control and repeatability but may miss context-dependent faults.ODC classifies defect semantics, G-SWFIT derives operators from field defects, and ProFIPy makes such catalogs programmable.Source: A. Software Fault Injection · p. 2

B. Learned and Generative Fault Models

Prior learned mutation work emphasizes code-level metrics rather than deployed-system consequences. This study compares accepted generative and catalog-based mutations in a multi-service cloud, using LLMs fine-tuned on residual Python bugs.

B. Learned and Generative Fault Models

  • Prior learned and generative methods emphasize syntax, historical similarity, mutant survival, or mutation score, which do not characterize operational consequences in a deployed system.Such methods can also produce invalid, equivalent, runtime-incompatible, or unrepresentative changes.Source: B. Learned and Generative Fault Models · p. 2
  • The study deploys accepted mutations in a multi-service cloud and observes effects across components and user-visible state, rather than evaluating the system’s test suite.It addresses the stated gap in comparisons of catalog-driven and generative source-level injection under common deployment, workload, and oracle configurations.Source: B. Learned and Generative Fault Models · p. 2

III. METHODOLOGY

  • The comparison evaluates fine-tuned LLMs and ProFIPy over shared target functions, using the ProFIPy testbed, workload, and four failure oracles.The evaluated LLMs are fine-tuned on PyResBugs, which contains 5,007 hand-validated residual bugs paired with fault-free functions and faulty versions left in production.Source: §III · p. 2, A. Dataset · p. 2

A. Dataset

  • Qwen and DeepSeek are trained on code pairs for direct clean-to-faulty transformations, and target functions and their bug-fixing commits are excluded from fine-tuning.A shared project or function name alone does not count as leakage; a match requires identical target code or fixing commit.Source: A. Dataset · p. 2

B. Fault Generation

The method contrasts LLM-generated candidates with ProFIPy’s controlled operator, then runs accepted mutations on a common testbed. Four oracles classify failure outcomes, while a separate dimension records propagation.

B. Fault Generation

  • Qwen2.5-Coder and DeepSeek-Coder are fine-tuned as comparable code-specialized models, while ProFIPy applies a controlled operator to generate mutations.The harness accepts Python 2.7-compatible mutations and runs them on the same testbed before classification using four oracles.Source: B. Fault Generation · p. 2, Figure 1 · p. 3
  • LLM candidates pass checks for a nonempty function definition and Python 2.7 compatibility before acceptance; candidates are not checked for semantic equivalence, signature preservation, or exactly one semantic edit.The first candidate passing both checks enters the campaign; targets failing all three attempts are dropped.Source: B. Fault Generation · p. 3

C. Measurement and Analysis

  • The API, state-assertion, log, and coverage oracles detect client-visible errors, inconsistent resource state, internal errors, and whether mutated code executes.Coverage first determines activation; API, assertion, and log observations then distinguish absorbed faults from manifest failures.Source: C. Measurement and Analysis · p. 3
  • The adapted CRASH scale classifies end-to-end failure severity, including Silent outcomes when an HTTP 2xx response leaves invalid or inconsistent state; no Restart occurred.Catastrophic denotes a service-daemon start-up crash, Abort an immediate API error, Hindering a late assertion followed by an API error, and Restart a hang or timeout requiring recovery.Source: C. Measurement and Analysis · p. 3, Table I · p. 3
  • The scenario inventory uses one mutated target per scenario.This bounds each scenario to a single target mutation.Source: Table II · p. 4
  • IMPACT separately classifies propagation as Total, Multi, Local, or Dormant, distinguishing severity from how far effects spread.Local includes activated runs confined to the injected component even when no failure is observed.Source: C. Measurement and Analysis · pp. 3–4

D. Experimental Design

The design evaluates Nova and Cinder under a shared OpenStack Pike testbed and workload, using coverage, API, assertion, and log observations to assess fault activation and outcomes.

D. Experimental Design

  • OpenStack Pike ran on CentOS 7 and Python 2.7, with controller services and a KVM/QEMU compute node; injection targeted Nova compute/driver and Cinder volume components.Source: D. Experimental Design · p. 4
  • The shared REST workload provisions a VM, network, and volume; checks SSH and block-device reads and writes; tests a hard reboot; then deletes resources.Source: D. Experimental Design · p. 4
  • Activation requires the workload to reach the target; an activated run with no API, assertion, or log trace may be absorbed, irrelevant, or equivalent, causes the experiment cannot distinguish.Source: D. Experimental Design · p. 4
  • The 122 one-target scenarios comprised 66 Nova and 56 Cinder runs, with a fault-free checkout after each injection to avoid environmental carryover.Source: D. Experimental Design · p. 4

IV. RESULTS

Across the compared failure profiles, LLM-generated faults more often produced Catastrophic or workflow-specific outcomes, while ProFIPy produced more Silent failures and propagated effects; shared-target activation was similar.

in the failures they expose?

  • 34/37 ProFIPy, 36/37 DeepSeek, and 37/37 Qwen shared-target faults activated; observable failures occurred in 19/37, 18/37, and 22/37 runs, respectively.Source: in the failures they expose? · p. 4
  • On Nova, Catastrophic outcomes were 28.6% for DeepSeek and 22.7% for Qwen versus 4.3% for ProFIPy; Silent outcomes were 30.4%, 9.5%, and 18.2%, respectively.Source: in the failures they expose? · p. 4
  • Cinder produced no Catastrophic outcomes; Hindering failures were 20.0–26.3%, while runs with no observed failure were 47.4–75.0%, and exception handling often contained injected errors.Source: in the failures they expose? · pp. 4–5
  • On Nova, dormant faults accounted for four ProFIPy, one DeepSeek, and one Qwen run; activated faults without observed failure could reflect absorption, workload scope, or semantic equivalence.Source: in the failures they expose? · p. 5
  • A Qwen mutation let a volume-attachment request return HTTP 2xx despite an unusable volume, detected only by a subsequent in-guest write/read assertion.Source: in the failures they expose? · p. 5
  • Failure manifestation depended on service architecture: Nova showed outcomes from start-up crashes to silent corruption, while Cinder more often contained or delayed effects.Source: in the failures they expose? · p. 5
  • On shared Nova targets, LLM-generated faults produced more Catastrophic and workflow-specific outcomes, while ProFIPy produced more Silent failures and propagated effects; aggregate activation differences partly reflect target selection.Source: in the failures they expose? · p. 5

propagation across system components?

Fault effects differed in propagation scope as well as severity: ProFIPy more often affected multiple components, while LLM-generated effects were more often local or total.

propagation across system components?

C. RQ3: How much does the choice of the generative model

Qwen and DeepSeek often produce different failure manifestations at the same injection locations, but agree more on how far effects propagate. Their observed profiles also vary by model and location, with mutation-style explanations remaining qualitative.

affect the operational behavior of injected faults?

  • On Nova, DeepSeek causes 6/21 Catastrophic failures (95% Wilson interval 14–50%), while Qwen causes 5/22 (10–43%).The intervals are wide and overlapping; Qwen produces the largest Abort share on Cinder (21.1%).Source: affect the operational behavior of… · p. 6
  • CRASH outcomes match at only 55% of shared Nova locations and 71% of shared Cinder locations, versus IMPACT agreement of 80% and 88%, respectively.The models often differ in failure manifestation at the same location, while propagation scope agrees more closely.Source: affect the operational behavior of… · p. 6
  • Cinder’s stronger error containment can reduce observable model differences, while highly vulnerable locations can dominate outcomes regardless of the generated mutation.Model-specific differences are more visible where execution continues and errors can follow different control-flow or state-propagation paths.Source: affect the operational behavior of… · p. 6
  • Manual inspection suggests DeepSeek often makes small interface-level changes, whereas Qwen more often rewrites function bodies or adds branches and exception handling.These mutation-style observations are qualitative because not every generated mutation received a systematic taxonomy.Source: affect the operational behavior of… · p. 6
  • Similar validity and activation rates do not imply equivalent operational behavior, although architecture and injection location remain major determinants of system-level impact.The models can produce different CRASH outcomes at the same target while showing higher agreement on IMPACT scope.Source: affect the operational behavior of… · p. 6

V. DISCUSSION

LLMs can generate context-dependent mutations without authoring a dedicated operator for each fault pattern, broadening the explored fault space. Their outputs still require controls for representativeness, mutation scope, validation, and reproducibility.

V. DISCUSSION

  • LLMs can broaden the explored fault space and make application-specific mutations easier to generate, but context-specific changes alone do not ensure representativeness.The process still needs mutation-scope control, evidence that modified code executes, representative workloads, system-level oracles, and reproducible provenance.Source: §V · p. 6

A. From Fixed to Data-Driven Fault Models

  • LLMs use local context to express defects that might require dedicated operators, but can introduce multiple semantic changes or replace too much of the implementation.For error-propagation studies, the paper suggests favoring single semantic changes and execution-preserving defects.Source: A. From Fixed to Data-Driven Fault… · p. 6
  • Figure 3 contrasts an LLM exception-handling rewrite with ProFIPy’s AST transformation that removes calls matching volume.The LLM uses surrounding exception-handling context; ProFIPy’s operator is precise and reproducible but does not account for each matched call’s semantic role.Source: A. From Fixed to Data-Driven Fault… · p. 6, Figure 3 · p. 7
  • The LLM’s context-specific rewrite needs no dedicated operator, but its generated change is not automatically atomic, representative, or reproducible.A realistic generative campaign therefore needs explicit controls on mutation scope and intent.Source: A. From Fixed to Data-Driven Fault… · p. 6
  • Validation should go beyond parsing or compilation to check deployability, mutation scope and interface, workload activation, and observable operational effect.The section points to these as distinct validation properties rather than treating compilable output as sufficient.Source: A. From Fixed to Data-Driven Fault… · pp. 6–7

B. Implications for Developers and Researchers

The discussion recommends placing LLM generation inside a staged fault-injection pipeline with deterministic validation and system observation. It also calls for runtime feedback, multi-level evaluation, reproducible provenance, and a hybrid path back to deterministic operators.

B. Implications for Developers and Researchers

  • A staged pipeline should let the model propose mutations while deterministic components enforce structure, runtime compatibility, deployment, activation checks, and system-level observation.This division retains generative flexibility without delegating experimental validity entirely to the model.Source: B. Implications for Developers and… · p. 7
  • Runtime feedback can guide regeneration of dormant mutations and balance immediate initialization failures with execution-preserving behavior.Failure observations can also prioritize mutations for campaign objectives such as recovery, silent-state corruption, containment, or cross-component propagation.Source: B. Implications for Developers and… · p. 7
  • Evaluations should measure activation, failure visibility and severity, propagation, and final system state, not only code-level properties.Comparisons should control injection location where possible because target code and architecture can dominate behavior.Source: B. Implications for Developers and… · p. 7
  • Run records should retain generation and validation decisions, deployment, coverage, workload observations, and failure classifications to distinguish generation variability from testbed or workload effects.The passage also specifies target location, prompt, and generation attempts as provenance to retain.Source: B. Implications for Developers and… · p. 7
  • A hybrid lifecycle can use LLMs for application-specific discovery, then review and encode recurrent or operationally relevant mutations as deterministic operators.This makes the rule-based catalog an evolving, reproducible core informed by generative exploration.Source: B. Implications for Developers and… · p. 7

VI. THREATS TO VALIDITY

The authors characterize the findings as evidence from a limited evaluated campaign, not universal superiority. Within that scope, LLM-generated faults broaden observed failure behavior, while realistic use still requires controls and validation.

VI. THREATS TO VALIDITY

  • Results are bounded by two OpenStack services, one version and workload, 122 partly overlapping scenarios, two code-specialized models, one dataset and generation configuration, and one rule-based injector.The authors caution that findings may differ across other services, workloads, concurrency levels, platforms, or fault locations.Source: §VI · p. 7

VII. CONCLUSION AND FUTURE WORK

  • In the evaluated OpenStack campaign, LLM-generated faults broadened the observed failure space, while ProFIPy provided greater control, repeatability, and more execution-preserving and propagating failures.The paper frames this as complementarity, not generator-level superiority or a universal result.Source: §VII · p. 8
  • Realistic generative fault injection still requires controlled mutation scope, deployment and activation validation, representative workloads, system-level oracles, and reproducible provenance.Future work includes broader settings, runtime feedback in generation, and promoting relevant generated faults into deterministic operators.Source: §VII · p. 8
Loading 2609.08681v1…