PaperlayerView exact revision on arXiv

Source-linked AI summary

Nuances are the Key: Unlocking ChatGPT to Find Failure-Inducing Tests with Differential Prompting

Tsz-On Li, Wenxi Zong, Yibo Wang, Haoye Tian, Ying Wang, Shing-Chi Cheung, Jeff Kramer

arXiv:2304.11686v6cs.SE

TL;DR

Finding failure-inducing tests requires fault-triggering inputs and correct assertions, yet direct ChatGPT use has limited success. Differential Prompting uses inferred program intention to generate reference versions for differential testing, and outperforms baselines on QuixBugs and Codeforces.

  • Problem

    Finding failure-inducing tests requires fault-triggering inputs and correct assertions, a difficult task involving vast search spaces and often-undecidable oracle construction.

    Source: §I · p. 1

  • Method

    Differential Prompting uses ChatGPT-inferred program intention to generate multiple compilable reference versions, which are then used for differential testing.

    Source: 1) Overview of Program Generator · p. 3, 2) Illustration of Program Generator’s… · p. 4

  • Results

    Differential Prompting outperforms the best baselines on QuixBugs and Codeforces; its success rate is 75.0% versus 28.8% on buggy QuixBugs programs.

    Source: §I · p. 2

  • Takeaways & Limitations

    The authors describe the evaluation as a step toward finding failure-inducing tests for larger real-world software.

    Source: 2) Generalizability · p. 10

    The evaluation covers only simple QuixBugs and Codeforces programs with fewer than 100 lines of code, limiting evidence about generalization to large real-world software.

    Source: 2) Generalizability · p. 10, E. Threat to Validity · p. 10

Abstract

from arXiv · show

Automatically detecting software failures is an important task and a longstanding challenge. It requires finding failure-inducing test cases whose test input can trigger the software's fault, and constructing an automated oracle to detect the software's incorrect behaviors. Recent advancement of large language models (LLMs) motivates us to study how far this challenge can be addressed by ChatGPT, a state-of-the-art LLM. Unfortunately, our study shows that ChatGPT has a low probability (28.8%) of finding correct failure-inducing test cases for buggy programs. A possible reason is that finding failure-inducing test cases requires analyzing the subtle code differences between a buggy program and its correct version. When these two versions have similar syntax, ChatGPT is weak at recognizing subtle code differences. Our insight is that ChatGPT's performance can be substantially enhanced when ChatGPT is guided to focus on the subtle code difference. We have an interesting observation that ChatGPT is effective in inferring the intended behaviors of a buggy program. The intended behavior can be leveraged to synthesize programs, in order to make the subtle code difference between a buggy program and its correct version (i.e., the synthesized program) explicit. Driven by this observation, we propose a novel approach that synergistically combines ChatGPT and differential testing to find failure-inducing test cases. We evaluate our approach on Quixbugs (a benchmark of buggy programs), and compare it with state-of-the-art baselines, including direct use of ChatGPT and Pynguin. The experimental result shows that our approach has a much higher probability (77.8%) of finding correct failure-inducing test cases, 2.7X as the best baseline.

I. INTRODUCTION °

The introduction frames finding failure-inducing tests as difficult because it requires both fault-triggering inputs and correct assertions. It presents Differential Prompting as a way to use inferred program intention and differential testing to find those tests.

Problem and Challenges

  • Finding useful tests requires both fault-triggering inputs and correct assertions; direct ChatGPT succeeded on 28.8% of buggy QuixBugs programs.The authors attribute this difficulty in part to LLM insensitivity to subtle code differences between buggy and fixed programs.Source: §I · pp. 1–2

Insights

  • 91.0% of buggy programs had their actual intention correctly inferred by ChatGPT, which the authors link to overlooking bug-level nuances.The authors’ insight is that this weakness can help ChatGPT infer intended behavior despite bugs.Source: §I · p. 2

Approach

  • Differential Prompting infers a program’s intention, generates alternative compilable programs with that intention, then uses differential testing to identify likely failure-inducing inputs.It considers only tests that produce consistent results across generated programs; if the program-under-test differs, the test is likely failure-inducing, assuming the generated programs do not commonly share its bug.Source: §I · p. 2

Evaluation

  • 75.0% of buggy QuixBugs programs and 66.7% of similarly complex buggy Codeforces programs yielded failure-inducing tests, versus best-baseline rates of 28.8% and 16.7%, respectively.For correct programs, the approach incorrectly suggested failure-inducing tests for 5.0% on QuixBugs and 1 out of 7 on Codeforces; both baselines did so for every correct program.Source: §I · p. 2

Contributions

  • The authors present this as the first study of ChatGPT for finding failure-inducing tests.They also suggest that such tests can help programming beginners detect failures and learn debugging practices.Source: §I · pp. 2–3

Promising Application Scenarios

  • Differential Prompting works best on simple programs with fewer than 100 lines of code; larger software brings harder-to-describe requirements and larger input spaces.The authors describe this improvement as a step toward finding failure-inducing tests for larger software.Source: §I · pp. 2–3

II. PRELIMINARIES

The preliminaries distinguish correct failure-revealing tests from coincidental or false failures, passing tests, and illegal-input tests. The distinctions support the paper’s criterion for evaluating failure-inducing tests.

Test-case categories

  • A correct failure-inducing test case (FT-IA) pairs an input that triggers an incorrect output with an assertion that correctly identifies that output.For example, program1(12,20) should return 4, so asserting 4 reveals the bug.Source: §II · p. 3
  • A coincidental failure-inducing test case (FT-Ia) has a valid failure-triggering input but an incorrect assertion, so its failure does not help diagnosis or repair.It can also reject correct patches.Source: §II · p. 3
  • A false failure-inducing test case (FT-ia) combines a non-failure-inducing input with an incorrect assertion, producing a false alarm.The example runs program1(17,0) and incorrectly asserts an output of 18.Source: §II · p. 3
  • A passing test (PT) passes, while an illegal argument test (IT) uses an input that violates the designated argument type.With a correct assertion and non-failure-inducing input, a test is either PT or IT; only FT-IA counts as a correct failure-inducing test case.Source: §II · p. 3

III. METHODOLOGY

Differential Prompting generates intention-based reference versions of a program-under-test (PUT), then uses them to support differential testing. Rather than requiring ChatGPT to generate a correct patch, it leverages inferred intention to create compilable alternatives.

A. Program Generator

  • Only 6.8% of ChatGPT programs prompted to be bug-free versions of QuixBugs programs were correct.This weak performance motivates the intention-based reference-version mechanism.Source: A. Program Generator · p. 3

1) Overview of Program Generator:

  • Program Generator first infers the PUT’s intention, then generates multiple compilable reference versions from that intention.This avoids relying on ChatGPT to generate a correct patched program from buggy code.Source: 1) Overview of Program Generator · p. 3
  • Two reference versions are the default, the minimum needed to validate an expected output; their number is configurable.The evaluation reports effective FT-IA finding with this default.Source: 2) Illustration of Program Generator’s… · p. 4

B. Test Case Generator

The Test Case Generator uses ChatGPT-generated inputs and intention-based reference versions to identify output differences from the PUT. It accepts only consistent reference outputs as expected results and stops when testing is adequate or its attempt limit is reached.

B. Test Case Generator

  • Test Case Generator finds failure-inducing tests by combining test-input generation, expected-output inference from reference versions, and differential testing.It is the second main component of Differential Prompting.Source: B. Test Case Generator · p. 4

1) Step 1: Generating test input:

  • ChatGPT generates diverse test inputs rather than inputs explicitly chosen to expose differences between the PUT and reference versions.The latter prompt would require ChatGPT to recognize subtle code differences, which the paper says it does poorly.Source: 1) Step 1: Generating test input · p. 4
  • A test input is discarded if it cannot trigger a PUT failure or produces inconsistent outputs across reference versions.The generator considers candidate inputs in turn until it finds a failure-inducing test case.Source: 1) Step 1: Generating test input · p. 4

2) Step 2: Inferring an expected output:

  • Only inputs producing the same output across all reference versions receive that output as the expected result.The strategy assumes the reference versions are sufficiently diversified, making a shared bug less likely.Source: 2) Step 2: Inferring an expected output · p. 4

3) Step 3: Differential testing:

  • A detected difference between PUT and reference outputs yields a test case using the selected input and inferred expected output.Testing stops when accumulated branch coverage reaches 100% or saturates; the generator then reports that no failure-inducing test case was found if none has been found.Source: 3) Step 3: Differential testing · p. 5

IV. EVALUATION

  • The evaluation frames four questions: finding QuixBugs failure-inducing tests, inferring intentions, generating reference versions, and finding tests for Codeforces programs.Experiments used ChatGPT API version gpt-3.5-turbo-0301 for reproducibility.Source: §IV · p. 5

A. RQ1: Finding FTs for QuixBugs

The QuixBugs evaluation compares Differential Prompting with direct ChatGPT prompting and Pynguin on Python programs. It uses repeated runs and distinct measures for correct failure-inducing tests and overall test-case accuracy.

1) Experiment setup:

  • BASECHATGPT prompts ChatGPT first to assess whether a program has bugs and then, if so, to generate a failure-inducing test case.PYNGUIN is the state-of-the-art unit-test generation tool for Python used as the second baseline.Source: 1) Experiment setup · p. 5
  • The study evaluates all 80 Python programs in QuixBugs: 40 buggy-and-patched pairs, each buggy program containing one bug.The programs implement common algorithms; the evaluation leaves QuixBugs Java programs for future work.Source: 1) Experiment setup · p. 5
  • The researchers remove QuixBugs comments and problem descriptions to avoid influencing program-intention inference.Those descriptions are treated as ground truth for the intention-inference evaluation.Source: 1) Experiment setup · p. 5
  • Ten runs are used for Differential Prompting and BASECHATGPT; Pynguin is run until ten test cases per experiment, with its top ten retained if it produces more.Pynguin's average failure-inducing-test count is rounded to an integer, while the success rate is calculated comparably across techniques.Source: 1) Experiment setup · pp. 5–6
  • Accuracy is the number of correct failure-inducing tests divided by all tests found for the targeted programs.The metric is adapted from precision to avoid confusion because Pynguin targets code coverage rather than failure-inducing tests.Source: 1) Experiment setup · p. 6

2) Results and findings:

Differential Prompting finds correct failure-inducing tests more often than the baselines on buggy QuixBugs programs and has higher accuracy across buggy and correct programs. Its remaining incorrect cases occur when generated reference versions share a bug.

2) Results and findings:

  • 75.0% success rate for buggy QuixBugs programs, 2.6X BASECHATGPT (28.8%) and 10.0X Pynguin (7.5%).These are the overall success-rate comparisons reported for the three techniques.Source: 2) Results and findings · p. 6, Table I · p. 6
  • 91.0% of buggy-program intentions are correctly inferred, and 74.6% of generated reference versions are good.The reported good-reference rate is for versions generated using the inferred intention.Source: 2) Results and findings · p. 6
  • Differential Prompting finds at least one correct test for 37 of 40 buggy programs and ten for 22; BASECHATGPT finds ten for only 2.Pynguin finds few correct tests; most of its test cases are PT and IT, and none of its PT contains a failure-inducing input.Source: 2) Results and findings · p. 6
  • When two generated reference versions share the same bug, Differential Prompting can return incorrect failure-inducing tests.The paper gives lcs-length and wrap as examples of this situation.Source: 2) Results and findings · p. 7
  • On correct programs, Differential Prompting returns incorrect failure-inducing tests for two programs, compared with all 40 for the baselines.The two affected programs are lcs-length and wrap, which also produce incorrect cases in their buggy versions.Source: 2) Results and findings · p. 7
  • 94.6% accuracy across buggy and correct programs, 6.6X the best baseline (14.3%).Accuracy on buggy programs alone is 98.0%, 3.4X the best baseline (28.8%).Source: 2) Results and findings · pp. 6–7, Table II · p. 7

B. RQ2: Inferring Program Intention

The authors manually assess ChatGPT’s inferred intentions against QuixBugs problem descriptions and find high overall accuracy. A few programs expose a limitation: inferred intentions can omit details essential to intended functionality.

1) Experiment setup:

  • The manual assessment treated each program’s documented problem description as ground truth and evaluated 400 intentions returned in RQ1.Two authors independently judged consistency with the descriptions and discussed discrepancies until reaching consensus.Source: 1) Experiment setup · p. 7

2) Results and findings:

  • 91.0% (364 out of 400) of inferred intentions were correct across 40 subjects.Two authors assessed the intentions against documented problem descriptions; their Cohen’s Kappa was 82.3%.Source: 1) Experiment setup · p. 7, 2) Results and findings · p. 8
  • Three subjects received only 1–3 correct intentions, with inferred intentions often describing the buggy implementation’s semantics rather than its intended functionality.For lcs-length, the inferred intention omitted that common substrings must be consecutive, leading to bad reference versions; the authors report similar cases for wrap and next_palindrome.Source: 2) Results and findings · p. 8

C. RQ3: Generating Reference Versions

Differential Prompting generates reference versions from inferred program intentions, and the authors evaluate whether those versions avoid the program’s bugs. It generates good reference versions more often than a direct-prompting baseline, though incorrect intentions limit performance on some subjects.

1) Experiment setup:

  • Each technique generated two reference versions per execution over ten executions for each of 40 buggy programs.The baseline directly prompted ChatGPT to generate bug-fixed implementations after asking whether the program had bugs.Source: 1) Experiment setup · p. 8

2) Results and findings:

  • 74.6% versus 6.8%: Differential Prompting’s success rate for good reference versions was 11.0X the baseline’s.A good reference version passes the failure-inducing test cases for its program; Figure 6 compares the techniques’ counts across subjects.Source: Figure 6 · p. 8, 2) Results and findings · pp. 8–9, 1) Experiment setup · p. 8
  • Differential Prompting generated 20 good reference versions for 20 subjects, while the baseline did so for only two; the baseline produced none for 34 subjects.In 89.2% of baseline executions, ChatGPT replied that no bug was found.Source: 2) Results and findings · p. 8
  • The authors attribute the baseline’s weaker performance to generating versions from the PUT’s implementation, whereas Differential Prompting uses its inferred intention.Their account is that intention-based generation bypasses ChatGPT’s difficulty identifying subtle code differences.Source: 2) Results and findings · pp. 8–9
  • Incorrect inferred intentions accounted for many bad versions on lcs_length, where 18 of 20 generated versions were bad.Providing the documented correct intention raised that subject’s good versions from 2 to 20 out of 20; the authors observed similar increases for next_palindrome and wrap.Source: 2) Results and findings · p. 9

D. RQ4: Finding FT-IA for Codeforces

The Codeforces evaluation uses selected Python buggy and correct program pairs from a post-training-cutoff contest. Differential Prompting outperforms the baselines on the selected buggy programs, while results indicate lower performance as program difficulty increases.

1) Experiment setup:

  • The evaluation selected Python buggy programs and their bug-fixed versions from Codeforces Round 835, held after ChatGPT’s September 2021 training cutoff.Pair selection required the correct program to be the bug-fixed version of the buggy program.Source: 1) Experiment setup · p. 9

2) Results and findings:

  • Differential Prompting performed best on the simplest selected program and its performance declined as program difficulty increased.The authors also report declining intention-inference performance as difficulty increased.Source: 2) Results and findings · p. 9, Figure 7 · p. 9
  • For Codeforces labels A–C, Differential Prompting’s success rate was 66.7%, compared with 75.0% on QuixBugs programs of similar complexity.Its accuracy for A–C was 100%, comparable to over 94.6% on QuixBugs.Source: 2) Results and findings · pp. 9–10
  • 41.0% versus 7.0%: Differential Prompting’s success rate on buggy Codeforces programs was 5.9X the best baseline’s.The comparison is reported for seven selected programs; Pynguin found no correct failure-inducing test cases in any executions.Source: 2) Results and findings · p. 9, Table III · p. 10
  • The authors caution that Differential Prompting may not work well on complex programs and suggest exploring reduction to simpler code snippets.The selected programs with labels D–G involve more advanced data structures or algorithmic techniques than A–C.Source: 2) Results and findings · pp. 9–10

E. Threat to Validity

The authors identify possible data leakage as a threat to the evaluation: if QuixBugs appeared in ChatGPT’s training data, measured performance could be overestimated. They assess this threat as likely insignificant and add post-cutoff Codeforces experiments.

1) Data leakage:

  • If QuixBugs was included in ChatGPT’s training data, evaluation results for both BASECHATGPT and Differential Prompting may be overestimated.QuixBugs is a public benchmark released in 2017, while ChatGPT’s training cutoff was September 2021.Source: 1) Data leakage · p. 10
  • The authors consider the leakage threat likely insignificant given BASECHATGPT’s 28.8% success rate and also evaluate post-cutoff Codeforces programs.Differential Prompting’s Codeforces performance was comparable with its QuixBugs performance.Source: 1) Data leakage · p. 10

2) Generalizability:

Evaluation is limited to QuixBugs and Codeforces programs under 100 lines, and the authors identify branch-specific bugs and larger programs as challenges for future work. They suggest test-coverage guidance and adapting Differential Prompting to coverage-based test generation.

2) Generalizability:

  • Evaluation covers only QuixBugs and Codeforces programs, all under 100 lines, limiting evidence about generalization to large real-world software.The authors note that large software often comprises small programs, but present this as potential rather than demonstrated generalizability.Source: 2) Generalizability · p. 10
  • Results use gpt-3.5-turbo-0301 and may not generalize to other LLMs such as GPT-4.The authors also warn that results may not be reproducible after the model is deprecated, while arguing the findings can remain useful references.Source: 2) Generalizability · p. 10, 3) Reproducibility · p. 10

V. DISCUSSION AND FUTURE WORK

  • Differential Prompting has a notably higher success rate than baselines, but missed bugs often lie on branches reachable only with specific test inputs.The authors suggest adding test-coverage guidance to direct generation toward uncovered statements.Source: §V · p. 11
  • Differential Prompting can be adapted to coverage-based test generation by using diverse generated tests and constructing their oracles with reference versions.The authors also suggest conversationally providing ChatGPT with uncovered statements to guide input generation.Source: §V · p. 11
  • A proposed future direction is to decompose large programs into small programs and infer failure-inducing tests from tests found for those smaller programs.Source: §V · p. 11

A. Finding failure-inducing test inputs

Differential Prompting is presented as a ChatGPT-based paradigm that significantly outperforms state-of-the-art baselines in finding failure-inducing test cases. The paper distinguishes it from related approaches by its assumptions about available information and its use of existing LLMs, and provides a reproduction package.

A. Finding failure-inducing test inputs

  • Unlike failing-test reproduction and augmentation, Differential Prompting assumes only a program-under-test, without knowing whether it is buggy or having bug reports or failing tests.Fuzzing is described as targeting predefined failure types and oracles, whereas Differential Prompting is not restricted to specific failure types or pre-defined oracles.Source: A. Finding failure-inducing test inputs · p. 11

B. Addressing Test Oracle problem

  • Differential Prompting and SEER are orthogonal: it uses existing commercial LLMs rather than training new models to address the test-oracle problem.Metamorphic testing and differential testing are cited as other paradigms for automated oracle construction, with often limited application scenarios.Source: B. Addressing Test Oracle problem · p. 11

C. Studies of ChatGPT and other LLMs

VII. CONCLUSION

  • Differential Prompting significantly outperforms state-of-the-art baselines in finding failure-inducing test cases.The paper presents it as the first paradigm for this task using ChatGPT.Source: §VII · p. 11

VIII. DATA AVAILABILITY

  • The reproduction package includes 470 inferred intentions, 940 reference versions, found failure-inducing test cases, a tool, and seven Codeforces programs.Source: §VIII · p. 11
Loading 2304.11686v6…