PaperlayerView exact revision on arXiv

Source-linked AI summary

An Empirical Evaluation of Using Large Language Models for Automated Model-Based Test Generation

Hafize Sanli, Onur Kilincceker, Cihat Cetinkaya

arXiv:2608.27094v1cs.SE

TL;DR

Industrial adoption of model-based testing is constrained by scalability, motivating evidence on whether LLMs can generate compact tests. The paper evaluates LLM4MBT across five LLMs and four GraphWalker models against GraphWalker baselines. LLM4MBT achieves comprehensive coverage with substantially fewer average test steps, while the authors identify coverage validity and stochasticity as limitations.

  • Problem

    Scalability challenges limit industrial adoption of MBT, motivating evaluation of shorter tests that preserve coverage.

  • Method

    The study evaluates LLM4MBT by testing five LLMs on four escalating-complexity GraphWalker models and comparing generated suites with GraphWalker algorithms.

  • Results

    96.3% average edge coverage and 100.0% average vertex coverage were achieved by LLM4MBT.

  • Takeaways & Limitations

    LLM4MBT shows potential to optimize and shorten test paths and step sizes across web and hardware GraphWalker models.

  • Takeaways & Limitations

    Coverage metrics do not always correlate with finding real-world bugs because weak oracles can permit 100% coverage without detecting logic flaws.

Abstract

from arXiv · show

Large language models have shown strong potential for software engineering tasks, particularly software testing. Model-based testing (MBT) is a software testing technique. To address the broad scalability challenge for industrial adoption of MBTs, our paper presents an empirical evaluation of Large Language Models (LLMs) for automated model-based test generation, compared with a state-of-the-art model-based testing tool (GraphWalker) and its built-in algorithms (random and quick random for edge and vertex coverage settings). Our evaluation indicates strong potential to optimize and shorten test paths and step sizes using the recent five state-of-the-art LLMs (GPT-5.1, GPT-5.2, Claude Opus 4.5, Claude Sonnet 4.5, and Gemini 2.5 Pro) against four GraphWalker models (two web applications (Parabank and Testinium) and two hardware applications (TLC and RISC-V) ) of escalating complexity.

1. Introduction

MBT uses models to describe expected system behavior and relies on automated test generation, but scalability remains a challenge for industrial adoption. This study evaluates whether LLMs can generate shorter tests than GraphWalker while supporting model-based testing.

  • Background: MBT uses models to describe the expected behavior of the system under test and supports abstract or directly executable automated test cases.Automated generation is a fundamental part of MBT and has been applied across web, mobile, and embedded systems.
  • Motivation: Scalability challenges hinder industrial MBT adoption, while compact test cases that preserve coverage can improve scalability.Prior test-reduction work motivates examining shorter tests rather than assuming longer tests are preferable.
  • Study objective: The study empirically assesses LLMs for automated model-based test generation against GraphWalker and its built-in algorithms, focusing on shorter tests.The comparison targets the state-of-the-art GraphWalker baseline and its traversal algorithms.
  • Contributions: The paper presents the LLM4MBT automation pipeline and evaluates recent LLMs on four realistic web and hardware systems.The evaluation addresses five research questions and includes a publicly shared dataset, implementation, and replication instructions.

2. Approach

The LLM4MBT approach evaluates whether structured prompts can induce valid coverage-targeted test suites from LLMs across GraphWalker models of escalating complexity. Its pipeline separates inputs, generation, processing, and reporting while validating paths and measuring coverage.

  • Approach: LLM4MBT evaluates whether LLMs can use software-testing knowledge and GraphWalker specifications to generate valid paths meeting coverage criteria.The approach contrasts LLM-generated paths with algorithmic or heuristic traversal approaches.
  • Experimental design: The study tests five LLMs against four GraphWalker models, ranging from a 10-vertex TLC to Testinium with 129 vertices and 259 edges.A single structured prompt is evaluated for producing syntactically and logically valid test suites.
  • Pipeline architecture: The pipeline comprises Input, Test Generation, Processing, and Evaluation & Reporting layers following C4 architecture principles.This modular separation supports systematic evaluation and later extension.
  • Test Generation Layer: The Test Generation Layer combines five LLMs, three strategies, and four models to produce 60 distinct test suites.Each LLM receives identical graph-model inputs and strategy-specific prompts.
  • Processing Layer: The execution engine validates transitions and restarts from stable vertices after invalid transitions, preserving valid segments for partial-success coverage measurement.This fallback mechanism prevents an invalid transition from terminating the entire suite.
  • Coverage analysis: Edge Coverage measures traversed edges and Vertex Coverage measures visited vertices, with success defined as EC ≥70% OR VC ≥85%.The metrics provide complementary views of transition testing and state-space exploration.

3. Research Questions and Evaluation Setup

The evaluation asks how LLM4MBT performs on coverage, compares with GraphWalker test-step length, examines prompt components, and assesses dependence on the underlying LLM. It uses five LLMs and four realistic models spanning embedded and web domains.

  • Evaluation of Coverage: RQ1 measures LLM4MBT vertex and edge coverage against 100% coverage targets, while acknowledging that failing paths may prevent LLMs from reaching 100%.GraphWalker is configured with 100% edge and vertex coverage targets as the baseline setting.
  • Comparative Analysis: RQ2 compares LLM4MBT coverage and test-step length with GraphWalker, motivated by the expectation that LLM-generated tests will be shorter.The comparison emphasizes coverage alongside total test-step length.
  • Ablation Study: RQ3 examines how specific prompt components influence the effectiveness and individual contributions of LLM-generated test steps.The ablation study varies prompts to quantify their contributions to overall performance.
  • Model Dependency: RQ4 assesses the degree to which achieved coverage correlates with the capabilities of the underlying LLM.The analysis quantifies variation across the five evaluated LLMs.
  • Evaluation Setup: The setup covers four realistic GraphWalker models across embedded systems and web applications, including TLC, RISC-V, Parabank, and Testinium.The evaluated LLMs are GPT-5.1, GPT-5.2, Claude Opus 4.5, Claude Sonnet 4.5, and Gemini 2.5 Pro.

4. Evaluation Results

Across four GraphWalker projects, LLM4MBT generated test suites with high coverage and fewer steps than GraphWalker baselines. Results also varied by model provider and prompt design, with Claude models showing the most consistent performance.

  • RQ1: LLM4MBT’s Coverage: LLM4MBT achieved 100.0% average vertex coverage and 96.3% average edge coverage across four GraphWalker projects.
  • RQ2: LLM4MBT vs. GraphWalker: Claude Opus 4.5 with Quick Random contributed an average of 29.7% relative to GraphWalker’s best-performing Quick Random algorithm.
  • RQ2: LLM4MBT vs. GraphWalker: 96.3% average edge coverage required 300 average test steps, versus 100.0% coverage requiring 6916 Random or 524 Quick Random steps.This corresponded to 23.0x and 1.7x reductions in test-suite size relative to the two GraphWalker strategies.
  • RQ3: Prompt Effect: Prompt engineering affected generation: terminology precision, explicit constraints, provider-specific instructions, and batching choices changed success and output quality.Using “quick_random” improved success rates; GPT models required more explicit instructions and per-model prompts than Claude models.
  • RQ4: Effect of Different LLMs: Claude Opus 4.5 achieved 100% edge coverage on RISC-V and Parabank under Quick Random, while Gemini 2.5 Pro failed completely on RISC-V.
  • RQ4: Effect of Different LLMs: Claude models were the most reliable across models and strategies, whereas Gemini 2.5 Pro showed model-specific strengths without generalization.

5. Threats To Validity

The evaluation faces threats from stochastic generation, prompt sensitivity, coverage metrics, model scope, and modeling-language variation. The authors describe mitigation strategies, while acknowledging boundaries on reproducibility, defect detection, and generalizability.

  • Internal Validity: LLM and GraphWalker randomness can produce different test paths across runs, limiting direct reproducibility.The authors propose multiple iterations or seeds as mitigation, but these were not fully applied.
  • Internal Validity: Prompt structure can substantially influence coverage, so findings may depend on the prompt-engineering strategies used.An ablation study with five state-of-the-art LLMs was conducted to mitigate this threat.
  • Construct Validity: High vertex or edge coverage may not correlate with finding real-world bugs when test oracles are weak.The authors plan mutation testing at model and code levels and intend to integrate test execution into the pipeline.
  • External Validity: Results from four GraphWalker models across hardware and web domains may not generalize to much larger industrial systems or highly nondeterministic behavior.The evaluated models were Traffic Light, RISC-V, Parabank, and Testinium.
  • External Validity: LLM4MBT effectiveness may vary across modeling languages such as Statecharts, Petri Nets, and BPMN.The study uses GraphWalker as a widely used model-based testing tool to mitigate this threat.

6. Related Work

Prior work establishes LLM potential in software testing, but systematic evaluation of recent LLMs for graph-based MBT across web and hardware applications remains underexplored. This study introduces LLM4MBT to address that gap.

  • LLMs in Software Testing: LLMs have progressed from code assistants toward broader software-development and software-testing roles, including autonomous test agents and intelligent oracles.Related work describes significant productivity gains in software testing and a shift away from manual scripting.
  • Research Gap: No prior study had systematically evaluated recent LLMs for automated MBT across web and hardware applications using graph-based specifications.The underexplored capability concerns navigating complex graph structures while achieving optimized coverage.
  • Research Gap: LLM4MBT is introduced to investigate whether LLMs can generate valid graph-based test paths that meet specified coverage criteria.The framework targets a gap left by evidence from unit testing and general industrial processes.

7. Conclusion

The paper evaluates LLMs for automated MBT generation against GraphWalker algorithms to address MBT scalability. Across five recent LLMs and four increasingly complex models, the evaluation indicates potential to shorten test paths and step sizes.

  • Conclusion: The evaluation compares five recent LLMs with GraphWalker’s random and quick-random algorithms for automated model-based test generation.The comparison covers edge- and vertex-coverage settings.
  • Conclusion: The evaluated systems include GPT-5.1, GPT-5.2, Claude Opus 4.5, Claude Sonnet 4.5, and Gemini 2.5 Pro across four GraphWalker models.The models represent two web applications and two hardware applications with escalating complexity.
  • Conclusion: The evaluation indicates strong potential for LLMs to optimize and shorten test paths and step sizes.This outcome is framed in relation to the four GraphWalker models and their built-in algorithms.

Declaration on Generative AI

The authors disclose using Grammarly and Bing Microsoft Translator for grammar and spelling checks. They state that the content was reviewed and edited and that they retain full responsibility.

  • Declaration on Generative AI: Grammarly and Bing Microsoft Translator were used to check grammar and spelling.The disclosure identifies these tools as preparation aids.
  • Declaration on Generative AI: The authors reviewed and edited the content as needed after using these tools.The statement describes author review of the resulting publication content.
  • Declaration on Generative AI: The authors assume full responsibility for the publication’s content.This is stated as the responsibility declaration accompanying the tool-use disclosure.
Loading 2608.27094v1…