Source-linked AI summary
ChatGPT and Software Testing Education: Promises & Perils
Sajed Jalil, Suzzana Rafi, Thomas D. LaToza, Kevin Moran, Wing Lam
TL;DR
The paper addresses limited evidence about ChatGPT’s performance and educational implications in software testing. It empirically evaluates textbook questions under different prompting and response-analysis conditions, finding partial correctness in answers and explanations and marginal benefits from shared context.
Problem
Evidence was lacking about ChatGPT’s capabilities, strengths, weaknesses, and educational implications in traditional instructional settings.
Method
The study evaluates ChatGPT on questions from the first five chapters of a widely used software testing textbook using multiple prompting strategies and repeated responses.
Results
55.6% of cases had correct or partially correct answers and 53.0% had correct or partially correct explanations; shared context produced marginally better answers.
Takeaways & Limitations
The findings indicate potential classroom uses for carefully designed activities while raising concerns about assessment circumvention and unreliable self-confidence.
Takeaways & Limitations
The study is limited to a dataset from the first five chapters, which may not cover all domains required for a software testing course.
Abstract
from arXiv · showhide
Over the past decade, predictive language modeling for code has proven to be a valuable tool for enabling new forms of automation for developers. More recently, we have seen the advent of general purpose "large language models", based on neural transformer architectures, that have been trained on massive datasets of human written text spanning code and natural language. However, despite the demonstrated representational power of such models, interacting with them has historically been constrained to specific task settings, limiting their general applicability. Many of these limitations were recently overcome with the introduction of ChatGPT, a language model created by OpenAI and trained to operate as a conversational agent, enabling it to answer questions and respond to a wide variety of commands from end users. The introduction of models, such as ChatGPT, has already spurred fervent discussion from educators, ranging from fear that students could use these AI tools to circumvent learning, to excitement about the new types of learning opportunities that they might unlock. However, given the nascent nature of these tools, we currently lack fundamental knowledge related to how well they perform in different educational settings, and the potential promise (or danger) that they might pose to traditional forms of instruction. As such, in this paper, we examine how well ChatGPT performs when tasked with answering common questions in a popular software testing curriculum. Our findings indicate that ChatGPT can provide correct or partially correct answers in 55.6% of cases, provide correct or partially correct explanations of answers in 53.0% of cases, and that prompting the tool in a shared question context leads to a marginally higher rate of correct responses. Based on these findings, we discuss the potential promises and perils related to the use of ChatGPT by students and instructors.
I. INTRODUCTION
The paper examines ChatGPT’s applicability to software testing education amid uncertainty about its capabilities and effects on instruction. It evaluates correctness, explanations, prompting strategies, and confidence using textbook questions and a manually vetted dataset.
- I. INTRODUCTION: ChatGPT’s conversational interface broadened interaction beyond the specific task settings that had constrained earlier language models.The model was trained using reinforcement learning from human feedback to refine conversational responses.
- I. INTRODUCTION: The paper frames ChatGPT’s educational use as involving both opportunities for new learning activities and concerns about circumventing learning.The authors emphasize that understanding capabilities, strengths, and weaknesses is necessary for informed educational decisions.
- I. INTRODUCTION: 55.6% of examined cases received correct or partially correct answers, while 53.0% received correct or partially correct explanations.ChatGPT properly responded to 77.5% of the testing questions examined.
- I. INTRODUCTION: The dataset contains manually vetted responses to 31 questions from five chapters of a popular software testing textbook.ChatGPT generated three responses for each question.
- I. INTRODUCTION: The study investigates how prompting strategies affect answer and explanation correctness, including shared versus separate question contexts.It also examines whether ChatGPT’s confidence level relates to correctness.
A. Chapters 1& 2 - Fault, Error and Failure
The first chapters define faults, errors, and failures, then connect testing practices such as RIPR, continuous integration, and test-driven development to detecting or preventing incorrect behavior.
- A. Chapters 1& 2 - Fault, Error and Failure: A fault is a static software defect, an error is an incorrect internal state, and a failure is externally observable incorrect behavior.These distinctions relate defects, internal program states, and behavior against requirements.
- A. Chapters 1& 2 - Fault, Error and Failure: The RIPR model requires reachability, infection, propagation, and revealability for a test to expose a failure.A test must reach defective code, produce an incorrect state, propagate it to the final state, and make it observable.
- A. Chapters 1& 2 - Fault, Error and Failure: Continuous integration repeatedly rebuilds code and reruns tests in a fresh environment after changes, helping identify introduced failures quickly.The process begins by obtaining code and tests, building the code, and running the tests.
- A. Chapters 1& 2 - Fault, Error and Failure: Test-driven development writes tests before implementation and repeats short cycles of failing tests, code changes, and additional tests.The methodology aims to produce clean, failure-free code by writing code under test to make tests pass.
D. Chapter 5 - Coverage Criterion and Subsumption
This section introduces coverage criteria and subsumption, then situates them within a dataset of software testing questions drawn from the first five textbook chapters.
- D. Chapter 5 - Coverage Criterion and Subsumption: A coverage criterion is a rule or set of rules imposing test requirements on a test set.It defines what a test set must satisfy for coverage.
- D. Chapter 5 - Coverage Criterion and Subsumption: Criterion C1 subsumes C2 exactly when every test set satisfying C1 also satisfies C2.Subsumption expresses an implication between test requirements.
- D. Chapter 5 - Coverage Criterion and Subsumption: The study asks how ChatGPT’s correctness, answer-explanation consistency, non-determinism, and confidence relate across prompting strategies.These are stated as research questions RQ1 through RQ4.
- D. Chapter 5 - Coverage Criterion and Subsumption: Nine questions requiring non-text outputs were removed, leaving 31 questions comprising 27 sub-questions and four independent questions.Examples of excluded tasks include screen printouts, fetching projects, and setting up continuous integration servers.
B. Data Collection Tool
The data collection tool automated ChatGPT querying under interface rate limits, while the experiment compared shared and separate conversational contexts across repeated runs.
- B. Data Collection Tool: The open-source tool automatically queried ChatGPT, collected responses, and waited 10 seconds before the next question because the interface was rate-limited.An API was not publicly available during data collection.
- B. Data Collection Tool: In separate-context queries, each of the 27 sub-questions was treated independently, producing 93 responses across three runs.The four independent questions were also asked three times in the study.
- B. Data Collection Tool: In shared-context queries, related sub-questions were asked sequentially in one ChatGPT session so prior prompts, code, and answers remained available.A new chat context was opened for each multi-part question.
- B. Data Collection Tool: The experiment collected 81 sub-question responses and 12 independent-question responses across three runs.Independent-question responses were identical across shared and separate datasets; only sub-question responses differed.
3) Confidence Query:
The study collected repeated ChatGPT responses and evaluated answer and explanation correctness separately, using author adjudication for disagreements.
- 3) Confidence Query:: ChatGPT questions were posed three times for RQ1–RQ3, while confidence queries were collected only for the first shared-context response.The confidence prompt asked ChatGPT how confident it was that its previous response was correct.
- 3) Confidence Query:: Two authors independently labeled responses, with a third author resolving disagreements through discussion.The evaluators had graduate-level or teaching experience with the software testing course and textbook.
- 3) Confidence Query:: Responses were split into answer and explanation components, each labeled correct, incorrect, or partially correct.The labels were assigned relative to the question’s solution.
- 3) Confidence Query:: The textbook question example concerned whether satisfying a subsuming coverage criterion necessarily satisfies the subsumed criterion.ChatGPT’s response was judged incorrect in its answer but partially correct in its explanation.
IV. RESULTS
The results compare shared and separate prompting contexts and examine whether answer and explanation correctness align across shared-context iterations.
- IV. RESULTS: 49.4% of shared-context answers were correct and 6.2% were partially correct, compared with 34.6% and 7.4% in separate contexts.Shared context therefore produced fewer incorrect answers on average in the reported comparison.
- IV. RESULTS: Explanation accuracy was higher in the shared context than in the separate context.The authors report that shared context was better when focusing on fully correct answers and explanations.
- IV. RESULTS: Shared context produced correct or partially correct answers in 55.6% of cases and explanations in 53.0% of cases.These aggregate results summarize the shared-context correctness comparison.
- IV. RESULTS: 11.8% of responses had different degrees of correctness for their answers and explanations.For example, an answer could be correct while its explanation was not.
- IV. RESULTS: Table II organizes answer-explanation outcomes into nine combinations across answer and explanation correctness categories.The table covers three shared-context iterations.
C. RQ3: How does ChatGPT’s non-determinism affect its
The study examines response variability across repeated prompts and ChatGPT’s self-reported confidence for answers and explanations.
- C. RQ3: How does ChatGPT’s non-determinism affect its: 9.7% of questions had answer correctness vary between correct and incorrect, while explanation correctness varied for 6.5%.The authors attribute differing responses to the stochastic sampling process.
- C. RQ3: How does ChatGPT’s non-determinism affect its: ChatGPT’s confidence was queried after responses, with calibration defined as greater confidence for correct responses than incorrect ones.The study used four confidence keywords and displayed answer and explanation results separately.
- C. RQ3: How does ChatGPT’s non-determinism affect its: When highly confident, ChatGPT’s answers were incorrect about half the time, while explanations were incorrect twice as often as correct.For the “Confident” category, answers or explanations were at least three times as likely to be correct than incorrect.
- C. RQ3: How does ChatGPT’s non-determinism affect its: Self-reported confidence had little bearing on question correctness and did not appear particularly useful for software testing questions.The authors interpret this as indicating that ChatGPT was not well calibrated in this setting.
- C. RQ3: How does ChatGPT’s non-determinism affect its: The case study examined incorrect-answer characteristics, effects of additional information, and an inconsistent answer-explanation pair.Its goal was to gather insight into why ChatGPT was wrong and how prompting strategies might affect correctness.
A. Characteristics of Incorrect Answers from ChatGPT
The case study categorizes incorrect ChatGPT answers and identifies missing knowledge and wrong assumptions as recurring characteristics.
- A. Characteristics of Incorrect Answers from ChatGPT: 13 incorrect answers from the first shared-context iteration were categorized into three main reasons for incorrect responses.Figure 6 summarizes the categorization.
- A. Characteristics of Incorrect Answers from ChatGPT: ChatGPT appeared to lack needed software-testing knowledge, including definitions of fault, failure, and error.For Chapter 5 questions, it also seemed not to understand how to determine whether a test set satisfies a coverage criterion.
- A. Characteristics of Incorrect Answers from ChatGPT: ChatGPT could focus on an irrelevant part of a question and answer incorrectly because it made a wrong assumption about what mattered.The paper refers to a separate example illustrating this characteristic.
- A. Characteristics of Incorrect Answers from ChatGPT: Four cases involved both missing knowledge and wrong assumptions.One example combined an incorrect assumption about a program fault with missing knowledge about errors.
B. Effect of Additional Information on ChatGPT’s responses
Providing additional prompt information corrected ChatGPT’s diagnosis of the oddOrPos fault, changing an initially incorrect null-check explanation into the correct negative-odd-number diagnosis.
- Initial response: ChatGPT initially misidentified the oddOrPos fault as missing a null check and proposed an incorrect null-check fix.The actual fault concerns negative odd values, which the original condition fails to count.
- Additional information: The modified prompt explicitly stated that the fault and modification were unrelated to a null check.
- Corrected response: ChatGPT’s modified response correctly identified the logical error and recommended checking odd values with x[i] % 2 != 0 or positive values with x[i] > 0.
C. Answer Correct, Explanation Incorrect (AC-EIC) Example
An example shows that ChatGPT can produce a correct answer while giving an incorrect explanation of why no error-without-failure test case exists.
- Question: The example asks whether a test case can produce an error without producing a failure, with a hint to consider the program counter.
- Answer and explanation: ChatGPT correctly answered that no such test case is possible, but incorrectly attributed this to NullPointerException halting execution.
- Correct explanation: The textbook states that every input producing an error also produces a failure because error states cannot be repaired.
- Threats to validity: The study’s dataset may not cover all domains needed for a software testing class, despite using a widely adopted textbook and including its first five chapters’ questions.
- Threats to validity: ChatGPT’s behavior may differ across future iterations and repeated invocations can produce inconsistent responses.The researchers ran each question three times to reduce the effect of inconsistency.
- Threats to validity: The main results used limited prompt engineering, so differently designed prompts might produce more correct answers.
- Threats to validity: Because the questions were presented without chapter contents, ChatGPT might be correct more often when additional context is supplied.
VII. RELATED WORK
Prior work documents both the capabilities and practical limitations of language models for programming and education. Against this backdrop, the paper evaluates ChatGPT in software testing and identifies uneven performance across question types, limited self-assessment, and possible classroom uses.
- Prior LLM applications: LLMs have been applied to code generation, with AlphaCode and GitHub Copilot illustrating their use in programming tasks.Prior studies reported human-like code similarity, competitive-programming performance, and real-world programming applications.
- Student programming: Studies of student programming tasks found Codex could outperform many students, while identical prompts sometimes produced substantially different algorithms and code sizes.These findings concern introductory programming as well as more advanced data-structures and algorithms examinations.
- Practical limitations: Usability studies found no measurable productivity benefit in some settings and mixed productivity benefits for Copilot, despite users enjoying interaction with code-generation systems.These results indicate that model capability does not uniformly translate into faster or more correct programming.
- Early ChatGPT evaluations: Early ChatGPT studies found performance near or above passing thresholds in selected medical and law examinations.The cited studies reported near-passing performance on all three parts of the US Medical Licensing Exam and a low passing average grade of C+ in four law classes.
- Software-testing evaluation: In software testing, ChatGPT produced correct or partially correct answers in 55.6% of questions, with correctness varying from 83.3% for coding questions to 31.3% for combined questions.It also produced correct or partially correct explanations in 53.0% of cases, while shared-context prompting yielded marginally better answers and confidence had little bearing on correctness.