PaperlayerView exact revision on arXiv

Source-linked AI summary

Software Testing with Large Language Models: Survey, Landscape, and Vision

Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, Qing Wang

arXiv:2307.07221v3cs.SE

TL;DR

Software testing must ensure software quality and reliability, while LLM use introduces challenges because testing requires diverse inputs rather than only single correct code snippets. This paper reviews LLM use in software testing from testing-task and LLM perspectives, finding broad application across testing tasks but persistent coverage, evaluation, and deployment challenges.

  • Problem

    Software testing is central to software quality and reliability, while testing’s need for diverse inputs creates challenges beyond conventional code generation.

  • Method

    The paper comprehensively reviews relevant studies using inclusion and exclusion criteria, analyzing LLMs in software testing from software-testing and LLM perspectives.

  • Results

    LLMs have been applied across unit test case generation, test oracle generation, system test input generation, program debugging, and program repair; test case preparation and repair-related tasks are prominent.

  • Takeaways & Limitations

    The review provides a roadmap identifying research gaps and opportunities for researchers and practitioners seeking to improve software testing practices.

  • Takeaways & Limitations

    LLMs have not been used in early testing stages such as test requirements and test planning, partly because these tasks are subjective and lack open-source data.

Abstract

from arXiv · show

Pre-trained large language models (LLMs) have recently emerged as a breakthrough technology in natural language processing and artificial intelligence, with the ability to handle large-scale datasets and exhibit remarkable performance across a wide range of tasks. Meanwhile, software testing is a crucial undertaking that serves as a cornerstone for ensuring the quality and reliability of software products. As the scope and complexity of software systems continue to grow, the need for more effective software testing techniques becomes increasingly urgent, making it an area ripe for innovative approaches such as the use of LLMs. This paper provides a comprehensive review of the utilization of LLMs in software testing. It analyzes 102 relevant studies that have used LLMs for software testing, from both the software testing and LLMs perspectives. The paper presents a detailed discussion of the software testing tasks for which LLMs are commonly used, among which test case preparation and program repair are the most representative. It also analyzes the commonly used LLMs, the types of prompt engineering that are employed, as well as the accompanied techniques with these LLMs. It also summarizes the key challenges and potential opportunities in this direction. This work can serve as a roadmap for future research in this area, highlighting potential avenues for exploration, and identifying gaps in our current understanding of the use of LLMs in software testing.

1 INTRODUCTION

Software testing remains essential but faces persistent challenges in coverage and meaningful test generation, creating opportunities for LLM-based approaches. This review analyzes 102 studies from software-testing and LLM perspectives, identifying common tasks, models, prompting strategies, accompanying techniques, challenges, and opportunities.

  • Motivation: Software testing supports software quality and reliability by finding failures, usability issues, and security breaches early.Early defect identification can also reduce maintenance costs and protect user trust.
  • Motivation: Existing automated testing techniques still produce unit tests whose coverage and meaningfulness are far from satisfactory.The introduction identifies search-based, constraint-based, and random-based approaches as examples.
  • Motivation: LLM-based testing differs from code generation because testing requires diverse inputs for coverage rather than a single correct code snippet.This distinction creates distinct challenges and opportunities for applying LLMs to software testing.
  • Review scope: The review collects and analyzes 102 relevant studies from both software-testing and LLM perspectives.It examines testing-task distributions, commonly used models, prompt engineering, LLM inputs, and accompanying techniques.
  • Findings: LLMs are commonly used for test case preparation, program debugging, and bug repair, but not for early testing tasks such as requirements and test planning.The review provides task-specific practices, technology trends, and reported performance.
  • Findings: About one-third of studies use pre-training or fine-tuning, while the remainder use prompting; zero-shot and few-shot strategies dominate, and traditional differential or mutation testing often accompanies LLMs.Chain-of-thought prompting and self-consistency are rarely used in the reviewed studies.
  • Challenges and opportunities: The field still faces challenges in testing coverage, test oracles, rigorous evaluation, and real-world application.Suggested opportunities include earlier lifecycle tasks, more software and non-functional testing, advanced prompting, and integration with traditional techniques.

2 BACKGROUND

LLMs are large, Transformer-based language models scaled through parameters, data, and computation, while software testing spans an iterative lifecycle from requirements through release. The paper frames these concepts and organizes testing activities across unit, integration, system, acceptance, and other testing types.

  • Large Language Models: LLMs are large language models trained on massive text data, typically using Transformer architectures and language-modeling objectives.The paper notes that no formal minimum parameter threshold exists and includes some smaller encoder-decoder and decoder-only models.
  • Large Language Models: Scaling model size, training data, and compute enables LLMs to better understand natural language and generate high-quality text from context or prompts.The paper discusses GPT-3, PaLM, Codex, and LLaMA as examples.
  • Software Testing: Software testing evaluates software quality and aims to identify defects that could cause incorrect or unexpected behavior.Its lifecycle includes requirements analysis, test planning, design and review, preparation, execution, reporting, bug fixing, regression testing, and release.
  • Software Testing: Testing is iterative and may repeat lifecycle stages according to software complexity and testing requirements.The lifecycle concludes with release after testing stages are passed and defects are fixed.
  • Testing Types: Testing types include unit, integration, system, acceptance, functional, performance, security, and accessibility testing.Unit testing targets components, integration testing combines modules, and system testing evaluates the integrated system and dependencies.

3 PAPER SELECTION AND REVIEW SCHEMA

The review combines broad database and venue searches with explicit selection, inclusion, exclusion, and quality-assessment procedures to identify relevant LLM software-testing studies.

  • Search strategy: The search covered four scientific databases and software-engineering and artificial-intelligence venues, using software-testing and LLM-related keywords.The databases were ACM Digital Library, IEEE Xplore, arXiv, and DBLP; manual searches covered selected major venues.
  • Study selection: The authors filtered the inclusive search pool using criteria requiring LLM-based software-testing approaches, practice applications, empirical studies, or specific testing techniques.Exclusions covered studies without software-testing tasks, without LLMs, or focused on other evaluation targets.
  • Study selection: The review excluded papers using LLMs only in future work, encoder-only language models, LLM-performance testing, or evaluation of LLM-enabled tools.The first two authors inspected papers and disagreements were referred to a third author for final decisions.
  • Final corpus: 47% of the papers appeared in software-engineering venues, while 46% were unpublished peer-reviewed papers disclosed on arXiv.Only 2% appeared in artificial-intelligence venues and 5% in program-analysis or security venues.

4 ANALYSIS FROM SOFTWARE TESTING PER-

The survey organizes studies by software-testing lifecycle tasks and finds LLM use concentrated in the middle and later stages, especially test preparation and bug handling.

  • General overview: The task analysis follows the software-testing lifecycle and notes that papers may belong to multiple task subcategories.Therefore, cumulative subcategory totals may not equal the number of papers.
  • General overview: LLMs have been used effectively in mid-to-late testing stages, including unit test generation, test-oracle generation, system test-input generation, test reporting, and bug fixing.Test-case preparation supports issue detection before further development, while later tasks address reports and fixes.

4.1 Unit Test Case Generation

Unit test case generation is a prominent LLM application, with research spanning tailored prompting, training, documentation, hybrid search, and comparative evaluation, but performance remains uneven.

  • Task and motivation: Unit test cases combine a test prefix that drives a focal method to a testable state with a test oracle that checks behavior.This distinguishes unit test generation from producing an isolated code snippet.
  • Task and motivation: Traditional unit-test generators use search-, constraint-, or random-based strategies, but generated-test coverage and meaningfulness remain unsatisfactory.LLM-based approaches were explored partly because code generation and unit-test generation both produce source code, while requiring task-specific treatment.
  • Approaches: Earlier studies commonly pre-trained or fine-tuned LLMs, whereas later work emphasized prompts containing parsed project information and adaptive focal contexts.Some studies continued training to improve domain familiarity.
  • Approaches: Additional techniques included mutation-informed prompts, security-test generation, API documentation for property-based tests, and hybrid search followed by LLM supplementation.The hybrid approach invokes the LLM after search-based coverage improvements stall to target under-covered functions.
  • Evaluation: Studies compared LLM-generated tests with EvoSuite or Pynguin using correctness, readability, coverage, and bug-detection capability.ChatGPT evaluations also reported compilation errors and execution failures in generated tests.
  • Evaluation: 2% coverage was reported for all three evaluated LLMs on the SF110 benchmark, although the study did not provide detailed reasons for this low performance.SF110 contains 111 open-source Java projects, 23,886 classes, over 800,000 bytecode-level branches, and 6.6 million lines of code.

4.2 Test Oracle Generation

LLM-based test oracle generation focuses mainly on producing test assertions, progressing from specialized pre-training and fine-tuning toward prompt-engineered few-shot methods.

  • Test oracle studies mainly target generating test assertions inside unit test cases.A test oracle determines whether software output is correct, while assertions indicate potential issues in tested code.
  • 17% of assertions from an earlier RNN-based approach exactly matched ground-truth assertions.
  • 57% exact-match performance was reported after T5 pre-training and fine-tuning, rising to 62% with separate English and code pre-training.
  • 76% exact-match accuracy was achieved with prompt-engineered few-shot code demonstrations, reported as state of the art for test assertion generation.The approach automatically retrieves similar demonstrations and evaluates different shot counts and prompt forms.

4.3 System Test Input Generation

LLMs are used to generate system-level test inputs across software types and testing techniques, with mobile apps prominent and fuzzing especially common.

  • Organization: System test input studies are organized by software type, testing technique, and input-output characteristics.Studies may appear in more than one subsection because these viewpoints are orthogonal.
  • Software types: Mobile-app input generation covers semantically meaningful text and GUI operations such as clicks and selections to expand functional and interface exploration.
  • Software types: LLMs support testing diverse software inputs, including DL programs constrained by programming-language syntax, semantics, APIs, and tensor shapes.
  • Software types: Mobile apps are the most prominent software category, with five studies using LLMs for testing.Other studied domains include deep learning libraries, compilers, SMT solvers, cyber-physical systems, and quantum computing platforms.
  • Testing techniques: Fuzz testing is the most commonly involved technique, using LLMs to generate diverse and realistic inputs that can improve testing effectiveness.
  • Testing techniques: Fuzz4All uses auto-prompting and an iterative LLM fuzzing loop across six languages, demonstrating higher coverage than existing language-specific fuzzers.
  • Testing techniques: Domain-specific fuzzers commonly combine LLM-based mutation with historical bug-triggering programs used for fine-tuning or demonstrations.

4.4 Bug Analysis

LLMs support bug analysis by enriching reports, triaging components, detecting duplicates, explaining bugs, and generating bug titles.

  • Bug-analysis studies categorize software bugs to improve understanding and facilitate subsequent debugging and repair.
  • LLMs generate follow-up answers for deficient reports and combine classification with generation for bug-component triaging.
  • Other applications extract information for duplicate-report detection, generate natural-language bug explanations, and automatically produce bug titles.

4.5 Debug

LLM-based debugging spans end-to-end diagnosis and repair, fault localization, bug reproduction, and error explanation.

  • Debugging is framed as identifying and locating a bug’s cause through code analysis, execution tracing, error information, and repair.
  • Overall debug framework: A unified Detect-Localize-Repair framework classifies buggy code, identifies faulty lines, and translates buggy code into a fixed version.
  • Bug localization: Fault-localization studies compare ChatGPT and GPT-4 with existing techniques while examining consistency, prompt engineering, and code-context length.
  • Bug reproduction: Bug-reproduction frameworks generate reproducing test cases or replay scripts, including mobile-app reproduction from bug reports through few-shot prompting and chain-of-thought reasoning.
  • Error explanation: LLMs are also used to explain compiler errors, including novice-focused explanations tailored to individual errors.

4.6 Program Repair

LLM-based program repair studies span bug types, repair settings, model-training strategies, and supporting techniques. The literature increasingly addresses multi-line repairs and prompt-based approaches, but benchmark scope limits conclusions about real-world effectiveness.

  • Program repair studies use LLMs to fix identified software bugs, exploiting their ability to process and analyze source code.
  • Repair research progressed from relatively simple single-line defects toward multi-line bugs involving semantic understanding, code-hunk modification, and refactoring.The shift reflects the observation that most real-world bugs involve multiple lines of code.
  • Fine-tuning was used in earlier multi-line repair studies, while later work emphasized prompt design and interactive feedback for patch generation.Prompt-based methods address fine-tuning's reliance on labeled data, computational resources, and susceptibility to overfitting.
  • Repair frameworks also combine LLMs with completion engines, mutation operators, static analyzers, multimodal prompts, iterative querying, and test feedback.Static-analysis-assisted repair first detects an error and then sends contextual information to the LLM to query a patch.
  • LLM repair commonly uses complete-function generation or code infilling, with QuixBugs and Defects4J among the most common datasets.Complete-function generation outputs an entire patch function, whereas code infilling fills a code chunk between a prefix and suffix.

5 ANALYSIS FROM LLM PERSPECTIVE

The review analyzes which LLMs, prompting strategies, inputs, and accompanying techniques are used in software-testing studies. ChatGPT leads model usage, prompt engineering is more common than weight updates, and many studies augment LLMs with testing techniques.

  • 5.1 LLM Models: ChatGPT is the most commonly used LLM in the collected software-testing studies, followed by Codex.CodeT5 ranks third, while 14 studies use GPT-4, ranking it fourth.
  • 5.2 Types of Prompt Engineering: 38 studies use pre-training or fine-tuning, whereas 64 use prompt engineering without updating model weights.
  • 5.2 Types of Prompt Engineering: Among prompt-engineering studies, 51 use zero-shot learning and 25 use few-shot learning; chain-of-thought appears in 7 studies.A study may use multiple prompt-engineering types; self-consistency and automatic prompting each appear in one study.
  • 5.2 Types of Prompt Engineering: Chain-of-thought prompting guides the LLM through intermediate reasoning steps before producing a final answer.Examples include bug localization, explanation, and repair, or generating a bug description before fuzzing code.
  • 5.2 Types of Prompt Engineering: Iterative prompt design refines prompts using testing-task feedback, including test-failure information during program repair.
  • 5.3 Input of LLM: Testing inputs include GUI view hierarchies and, with multimodal GPT-4, GUI screenshots can also be supplied directly.
  • 5.4 Accompanied Techniques: 67 studies use LLMs for the entire testing task, while 35 incorporate techniques such as mutation testing, differential testing, syntactic checking, or program analysis.

6 CHALLENGES AND OPPORTUNITIES

The survey finds that LLM-based software testing is growing but remains concentrated in limited testing phases, software types, and functional tasks. It identifies coverage, test oracles, evaluation rigor, and real-world deployment as major challenges, while outlining opportunities for broader coverage and integration with human and traditional techniques.

  • LLM-based software testing is still in its early development despite significant growth over the past two years.
  • Opportunities: Future work includes testing-specific model training, human-computer interaction for early testing, metamorphic oracles, multimodal interface analysis, and advanced prompt engineering.
  • Test oracles: Differential testing can mitigate the test-oracle problem by comparing multiple systems, but it requires suitable counterpart software or environments.
  • Real-world application: Real-world adoption is constrained because organizations often prefer open-source or medium-sized models that may underperform commercial systems and benchmark results.
  • Testing phases: The surveyed studies primarily use LLMs for unit and system testing, with no research identified for integration or acceptance testing.
  • Testing types: LLMs are mainly applied to functional testing, while performance and usability testing remain largely unexplored because they require specialized analysis.

7 RELATED WORK

Prior surveys covered broad software engineering, deep learning, computer vision, testing machine-learning systems, or general LLM topics. This paper specifically surveys LLM use in software testing and organizes its challenges and opportunities as a research roadmap.

  • Earlier surveys examined machine learning, deep learning, computer vision, or machine-learning-system testing within software engineering.
  • Other LLM reviews addressed general advances or broader software-engineering applications rather than focusing specifically on software testing.
  • This paper focuses specifically on LLMs for software testing, summarizes challenges and opportunities, and provides a roadmap for future research.

8 CONCLUSION

The paper reviews LLM use in software testing and concludes that applications span several testing tasks but remain concentrated in later-stage, unit/system, and functional testing. It presents the resulting gaps as directions for future research and practice.

  • The review finds successful LLM applications in unit test generation, test oracle generation, system test input generation, debugging, and program repair.
  • LLM use remains concentrated in middle and later testing stages, unit and system testing, and functional testing, leaving several areas uncovered.
  • The paper positions its findings as a roadmap for identifying knowledge gaps and future research opportunities in LLM-based software testing.

grated Venue Ref

The supplied passages are fragmented records from a table of collected studies, exposing partial titles, years, topics, models, prompting methods, inputs, venues, and techniques.

  • The records associate studies with venues or publication labels including EASE, ICSE, ISSTA, ASE, FSE, Arxiv, and SANER.
  • The records list prompting approaches including zero-shot learning, few-shot learning, fine-tuning, pre-training, and chain-of-thought prompting.
  • The records include study years, testing topics such as debugging, program repair, and system test input generation, and involved LLMs including ChatGPT, Codex, CodeT5, GPT-3, and GPT-4.
Loading 2307.07221v3…