Source-linked AI summary
Towards Autonomous Testing Agents via Conversational Large Language Models
Robert Feldt, Sungmin Kang, Juyeon Yoon, Shin Yoo
TL;DR
Software testing demands expertise and developer effort, motivating the search for LLM assistance. The paper proposes a taxonomy and a conversational testing-agent framework, then illustrates conversational testing with a Julia example in which execution feedback leads to revised test code.
Problem
Software testing takes substantial expertise and effort, and the paper considers how LLMs might help developers specify and test intended behavior.
Source: §I · p. 1Method
The paper proposes a taxonomy of LLM testing autonomy and SOCRATEST, a conversational framework that connects LLMs with testing tools through middleware.
Source: §III · p. 2Results
In a Julia clamp example, GPT-4 generated test suggestions, and after receiving execution results it corrected a mistaken test expectation and revised the code.
Source: §IV · p. 3Takeaways & Limitations
The authors argue that conversational testing can help developers refine test code and think more deeply about intended behavior, while sometimes making erroneous tests a prompt for reflection.
Source: §IV · p. 3Current LLMs lack agency to use external tools on their own, and their planning abilities are not well-defined.
Source: §VI · p. 4
Abstract
from arXiv · showhide
Software testing is an important part of the development cycle, yet it requires specialized expertise and substantial developer effort to adequately test software. Recent discoveries of the capabilities of large language models (LLMs) suggest that they can be used as automated testing assistants, and thus provide helpful information and even drive the testing process. To highlight the potential of this technology, we present a taxonomy of LLM-based testing agents based on their level of autonomy, and describe how a greater level of autonomy can benefit developers in practice. An example use of LLMs as a testing assistant is provided to demonstrate how a conversational framework for testing can help developers. This also highlights how the often criticized hallucination of LLMs can be beneficial for testing. We identify other tangible benefits that LLM-driven testing agents can bestow, and also discuss potential limitations.
I. INTRODUCTION
Software testing needs substantial expertise and effort, while test-writing can be repetitive. The paper proposes conversational LLMs as partners that can help translate developer intent into tests and clarify unexpected behavior.
I. INTRODUCTION
- Testing requires technical expertise, and automated test-generation tools can themselves be difficult to use in industry.Test-writing is also repetitive, while test syntax is often simpler than production-code syntax.Source: §I · p. 1
- A domain expert can specify project behavior while a testing expert turns those specifications into concrete tests.Their dialogue may surface unexpected specification issues and help clarify desired behavior.Source: §I · p. 1
- The authors propose that LLMs could supply testing knowledge, fill in lower-level details of developer intent, and use dialogue to offer counterexamples.With middleware connecting tools, an LLM could plan and use those tools toward a developer's high-level objective.Source: §I · p. 1
- The paper presents an autonomy taxonomy, a GPT-4 interaction example, potential benefits, and limitations of conversational testing agents.The example is intended to show how dialogue can prompt developers to consider fine-grained code semantics.Source: §I · p. 1
II. BACKGROUND
The background connects LLM-based agents and cognitive architectures to possible software-testing automation. It describes external tools, memory, and planning as capabilities relevant to testing agents.
II. BACKGROUND
- AutoGPT pursues user-defined goals with external tools and short- and long-term memory to handle tasks beyond current LLM context limits.The listed tools include search engines, other neural-network models, and file systems.Source: §II · p. 2
- AutoGPT's tools resemble perception and action components in cognitive architectures, while LLMs can propose actions from memory and current state.The paper names ACT-R and SOAR as influential architectures.Source: §II · p. 2
- LLM agent studies describe simulated social interactions with memory and reflection, and exploration of a 3D world without human guidance.The cited examples are Park et al. [15] and Wang et al. [16].Source: §II · p. 2
- For testing, the authors envision agents managing testing history, planning strategies and sub-goals, and evolving suites using testing tools and their results.They frame these capabilities as steps toward implementing a testing agent.Source: §II · p. 2
III. VISION - SOCRATEST
SOCRATEST is a proposed conversational testing-agent framework that connects LLMs to testing tools through middleware. Its taxonomy distinguishes autonomy levels and information sources, from single exchanges to agents that plan and execute tasks.
III. VISION - SOCRATEST
- SOCRATEST proposes middleware that invokes testing tools based on LLM output, with the aim of supporting conversational agents that can operate autonomously.The authors envision agents serving as human testing partners and handling typical testing tasks autonomously.Source: §III · p. 2
- The taxonomy's Driver column identifies who initiates tasks, gathers information, and chooses the next step; higher rows indicate greater LLM autonomy.For conversational testing, the human drives the next action through dialogue.Source: §III · p. 2, Table I · p. 3
- Autocompletion uses code context, while Contextual Prompting adds information such as examples but remains a single query-and-response without developer interaction.The paper notes that Contextual Prompting has supported tasks such as bug reproduction [17].Source: §III · p. 2
- Conversational Testing responds to human requests; tool-using systems can initiate tools under instructions, while testing agents add memory and planning to complete whole tasks.The proposed middleware can connect testing tools and augment LLMs with memory; the paper mentions LangChain as one possible framework.Source: §III · p. 2
IV. INSPIRATIONAL EXAMPLE TASKS
The authors illustrate conversational testing through a GPT-4 unit-testing interaction for Julia's clamp function. The exchange produced test suggestions, exposed a mistaken expectation about invalid bounds, and allowed the model to revise its code after receiving execution results.
IV. INSPIRATIONAL EXAMPLE TASKS
- GPT-4 first offered broad unit-testing advice, then focused on input selection with Equivalence Partitioning and Boundary Value Analysis.It combined the techniques into a checklist and generated 16 test cases for Julia's Base.clamp function.Source: §IV · p. 3
- The paper illustrates the generated tests with valid-input examples and an invalid-input check expecting an ArgumentError when lo exceeds hi.The later revised check instead asserts that clamp(5, 7, 3) equals 3.Source: §IV · p. 3
- Execution showed that 15 tests passed, but Julia returned 3 rather than throwing an ArgumentError when clamp(5, 7, 3) was called.GPT-4 then explained that Julia returns the hi value when it is lower than lo and updated the test code.Source: §IV · p. 3
- The interaction let the authors request more concrete recommendations and refine earlier test code, despite GPT-4 confidently reversing its earlier explanation.They describe potential pedagogical benefits and a reminder to apply testing techniques in new contexts.Source: §IV · p. 3
- The authors argue that an erroneous test expectation can prompt deeper thought about intended behavior, since test code is not deployed as part of the software.In this example, the mistaken expectation raised the question of whether invalid bounds should instead trigger an exception.Source: §IV · p. 4
V. PROGRESS TOWARDS VISION
The paper argues that LLM testing benefits grow with autonomy: hallucinated outputs may expose unexpected behavior, while tool-enabled agents could take on more of the testing workflow.
V. PROGRESS TOWARDS VISION
- LLM hallucinations can be useful in testing because generated tests may uncover unexpected software behavior.The authors say this potential applies across levels of LLM use for testing.Source: §V · p. 4
- Tool-enabled agents could translate natural-language testing guidelines into executable scripts and seek developer clarification when needed.The authors argue such guidelines could fit existing practices, improving acceptance of results and providing explanations.Source: §V · p. 4
- With greater autonomy, agents could formulate and execute testing plans, reducing manual execution effort and potentially using computing resources while developers are away.Source: §V · p. 4
VI. PRESENT-DAY LIMITATIONS
The paper identifies limits in current LLMs’ tool agency and planning abilities, alongside the costs and access constraints associated with training and operating them.
VI. PRESENT-DAY LIMITATIONS
- Current LLMs cannot use external tools on their own, although prompting techniques and systems can provide tool-produced information or tool access indirectly.Source: §VI · p. 4
- Current LLM planning abilities are not well-defined and are considered among their less-developed competencies.The authors note that multi-step prompting might mitigate this limitation, while hybrid systems using traditional AI planning tools may warrant exploration.Source: §VI · p. 4
- LLM training and operation costs drive energy requirements and limit access to resource-rich organizations, potentially impeding open-tool development.The authors also note that performance depends on training-data volume as well as model size, and that quantization and low-rank adaptation may help create smaller effective models.Source: §VI · p. 4
VII. CONCLUSIONS
The paper presents a taxonomy of conversational and potentially autonomous testing agents, an example interaction, and limitations that frame its vision. It identifies middleware and future tool-integration research as important to realizing that vision.
VII. CONCLUSIONS
- The paper presents a taxonomy of conversational and potentially autonomous testing agents and describes how greater autonomy could benefit developers.Source: §VII · p. 4
- An example LLM conversation provides initial confirmation that conversational testing can enhance developers’ testing effectiveness.Source: §VII · p. 4
- The authors identify middleware as critical to autonomous testing agents and plan to investigate software-engineering tools and their integration with LLMs.Source: §VII · p. 4