PaperlayerView exact revision on arXiv

Source-linked AI summary

An Empirical Study of the Non-determinism of ChatGPT in Code Generation

Shuyin Ouyang, Jie M. Zhang, Mark Harman, Meng Wang

arXiv:2308.02828v2cs.SE

TL;DR

Unstable code generation from identical prompts raises concerns about the validity and reproducibility of LLM-based software-engineering research. The paper empirically studies ChatGPT across 829 problems from three benchmarks, finding substantial non-determinism and concluding that researchers should account for it when drawing conclusions.

  • Problem

    Non-deterministic code outputs from identical prompts may undermine the reliability and reproducibility of empirical software-engineering research.

  • Method

    The study evaluates ChatGPT on 829 coding problems from CodeContests, APPS, and HumanEval, comparing five predictions per task across three similarity dimensions and experimental conditions.

  • Results

    Non-determinism is substantial across all three benchmarks, and temperature=0 does not guarantee deterministic code generation.

  • Takeaways & Limitations

    Researchers should take non-determinism into account when drawing conclusions from LLM-based code-generation experiments.

  • Takeaways & Limitations

    The findings may not generalize to other programming languages, similarity measures, prompts, or LLMs beyond ChatGPT.

Abstract

from arXiv · show

There has been a recent explosion of research on Large Language Models (LLMs) for software engineering tasks, in particular code generation. However, results from LLMs can be highly unstable; nondeterministically returning very different codes for the same prompt. Non-determinism is a potential menace to scientific conclusion validity. When non-determinism is high, scientific conclusions simply cannot be relied upon unless researchers change their behaviour to control for it in their empirical analyses. This paper conducts an empirical study to demonstrate that non-determinism is, indeed, high, thereby underlining the need for this behavioural change. We choose to study ChatGPT because it is already highly prevalent in the code generation research literature. We report results from a study of 829 code generation problems from three code generation benchmarks (i.e., CodeContests, APPS, and HumanEval). Our results reveal high degrees of non-determinism: the ratio of coding tasks with zero equal test output across different requests is 75.76%, 51.00%, and 47.56% for CodeContests, APPS, and HumanEval, respectively. In addition, we find that setting the temperature to 0 does not guarantee determinism in code generation, although it indeed brings less non-determinism than the default configuration (temperature=1). These results confirm that there is, currently, a significant threat to scientific conclusion validity. In order to put LLM-based research on firmer scientific foundations, researchers need to take into account non-determinism in drawing their conclusions.

1 Introduction

This study examines whether ChatGPT produces inconsistent code across requests with identical prompts and finds that the resulting non-determinism threatens empirical software-engineering conclusions. It evaluates this threat across benchmarks, settings, task features, and prompt strategies.

  • Motivation: Identical prompts can yield different code candidates because LLM generation uses probabilistic token sampling.The paper describes non-determinism as inconsistency across requests with identical prompts.
  • Motivation: Non-determinism threatens reliability and reproducibility because semantic inconsistency in generated code often indicates errors.The paper distinguishes code-related non-determinism as especially serious compared with some other ChatGPT tasks.
  • Study scope: 829 coding problems from CodeContests, APPS, and HumanEval are studied using five predictions per task and semantic, syntactic, and structural similarity measurements.The study also examines temperature, task-feature correlations, and prompt-engineering strategies.
  • Findings: More than 50% of problems have no equal test output among the top-five candidates across all studied benchmarks.The maximum test-pass-rate difference reaches 1.00 for all three datasets and occurs in 39.63% of HumanEval problems.
  • Findings: Only 21.1% of 76 surveyed LLM-based code-generation papers consider non-determinism in their experiments.The authors characterize this as a significant threat to scientific-conclusion validity and call for researchers to account for it.
  • Findings: Temperature 0 reduces non-determinism relative to the default setting but does not guarantee deterministic code generation.Longer coding instructions correlate negatively with similarity and average correctness, while prompt strategies produce different non-determinism levels.

2 Method

The experimental procedure repeatedly queries ChatGPT with the same coding prompt, extracts five code candidates, and compares their similarity. Prompts use zero-shot Python-generation instructions, with responses formatted for code extraction and compiled without modification.

  • Experimental procedure: Each task receives one coding prompt, which is sent to the ChatGPT API five times to produce five predictions.The study then extracts code from each response for comparison.
  • Experimental procedure: The five extracted code candidates are compared using semantic, syntactic, and structural similarity.These measurements operationalize the study’s analysis of non-determinism.
  • Prompt synthesis: Prompt preparation uses zero-shot Python generation with basic programming-task descriptions.The prompt design follows common practice in LLM-based code-generation assessment.
  • Code extraction: Responses specify Markdown formatting so generated code can be distinguished from plain text before extraction.The API response does not provide a clear code boundary comparable to Markdown code blocks in the web interface.
  • Code extraction: Extracted code is compiled directly without modifications in a Linux environment with required libraries pre-installed.This setup is intended to reduce import errors caused by missing libraries.

3 Experimental Design

The study evaluates ChatGPT’s non-determinism through multiple research questions covering similarity across repeated requests, temperature, task characteristics, models, and prompting. Experiments use established code-generation benchmarks and measure semantic, syntactic, and structural similarity with statistical analysis.

  • Research Questions: The study examines non-determinism across semantic, syntactic, and structural similarity, temperature, within-prediction comparisons, task characteristics, model versions, and prompt strategies.The research questions cover RQ1–RQ6, including temperature effects, task correlations, GPT-3.5 versus GPT-4, and prompt engineering.
  • Code Generation Benchmarks: The experiments use CodeContests, APPS, and HumanEval as code-generation benchmarks.The supplied passages identify these as widely studied benchmarks and describe their dataset-specific characteristics.
  • Configuration of ChatGPT: The study focuses on Python3 code generation and evaluates ChatGPT through the gpt-3.5-turbo and gpt-4 model APIs.ChatGPT is selected because of its adoption and performance in code-generation research, while Python3 is emphasized as a widely studied language.
  • Semantic Similarity: Semantic similarity is measured using test pass rate and output equivalence rate across generated code candidates.Output equivalence rate records the ratio of identical test outputs, while test pass rate measures the proportion of passed test cases.
  • Syntactic Similarity: Syntactic similarity uses Longest Common Subsequence and Levenshtein Edit Distance to compare candidate code texts.LCS measures normalized longest-common-subsequence length, whereas LED measures the minimum number of single-token edits.
  • Structural Similarity: Structural similarity compares candidates through Abstract Syntax Tree representations using pycode_similar and its United_Diff and Tree_Diff settings.The study also applies Kruskal-Wallis tests to analyze differences among outputs without assuming normally distributed data.

4 Results and Findings

Across semantic, syntactic, and structural measures, ChatGPT-generated code candidates vary substantially across repeated requests, with especially pronounced semantic differences. Structural similarity remains comparatively high, while syntactic similarity varies by benchmark.

  • Semantic similarity: 1.00 was the maximum test-pass-rate difference in all three datasets, occurring in 39.63% of HumanEval problems.Thus, correctness among candidates from the same instruction can differ substantially.
  • Semantic similarity: 92.1% of CodeContests, 39.4% of APPS, and 40% of HumanEval showed significantly different outputs under the Kruskal-Wallis test.The reported significance criterion was p-value < 0.05.
  • Semantic similarity: 75.76%, 51.00%, and 47.56% of CodeContests, APPS, and HumanEval tasks, respectively, had no equal test output among five candidates.These results indicate substantially different semantics across repeated requests for identical instructions.
  • Syntactic similarity: CodeContests was most syntactically unstable and HumanEval most syntactically stable across the three datasets.CodeContests had the lowest LCS and largest LED, whereas HumanEval had the largest LCS and smallest LED.
  • Syntactic similarity: The mean LCS was 0.22, 0.23, and 0.42 for CodeContests, APPS, and HumanEval, respectively.Lower LCS indicates lower syntactic similarity.
  • Structural similarity: Mean structural similarity was 0.33, 0.43, and 0.60 under United_Diff and 0.41, 0.54, and 0.62 under Tree_Diff for CodeContests, APPS, and HumanEval, respectively.Both structural measures were lowest for CodeContests and highest for HumanEval.

4.2 RQ2: Influence of Temperature

Lowering ChatGPT’s temperature improves determinism across semantic, syntactic, and structural similarities but does not eliminate non-determinism. Temperature 0 also preserves similar test-pass-rate values while reducing output variation relative to temperature 1.

  • Temperature effects: Temperature=0 produced better determinism than temperature=1 across all three similarity types, but did not completely avoid non-determinism.The experiment reports results for CodeContests, with other dataset results referenced separately.
  • Temperature effects: 43.64%, 27.40%, and 18.29% of CodeContests, APPS, and HumanEval problems, respectively, still had no equal test output at temperature=0.These values are reported for OER among five code candidates.
  • Temperature effects: Temperature=0.5 was more deterministic than temperature=1 but less deterministic than temperature=0.The paper attributes the pattern to higher temperature bringing more creativity and affecting code similarity.
  • Temperature effects: Test-pass-rate values were similar across temperatures, indicating that low temperature might be a better choice for code generation tasks.This conclusion accompanies the reported determinism differences.

4.3 RQ3: Non-determinism Comparison with Top Candidates in the Same Prediction

The study compares five top candidates generated across separate requests with five top candidates generated within one request. At temperature 1 their non-determinism is similar, whereas at temperature 0 the single-request configuration is more deterministic.

  • Configuration comparison: At temperature=1, it was difficult to determine which request configuration was more deterministic.Semantic similarity was similar across configurations, with only slight syntactic differences favoring the single-request configuration.
  • Configuration comparison: At temperature=0, candidates requested within one request showed higher determinism than candidates requested across separate requests.The single-request configuration had higher OER and LCS, lower LED, and closer-to-one structural similarity values.
  • Configuration comparison: Under default temperature, top-five candidates from one request had similar non-determinism to five top-one candidates from different requests.The distinction became more pronounced at temperature 0.

4.4 RQ4: Coding Tasks Features and Non-determinism Degree

Longer and more difficult CodeContests problems tend to produce more nondeterministic, syntactically and structurally varied, and bug-prone code. Examples and correlation analyses support this pattern, while the study explicitly avoids causal conclusions.

  • The authors analyze correlations rather than causal effects when relating coding-task characteristics to nondeterminism.
  • Longer coding-task descriptions tend to generate code with more randomness across most measurements, except LED.The authors suggest longer descriptions may reduce ChatGPT’s understanding of coding requirements.
  • Higher problem difficulty and CF rating correlate with higher syntactic nondeterminism, reflected by positive LED and negative LCS correlations.
  • A shorter description of length 1511 was associated with greater generation stability in a contrasting CodeContests example.Test pass rates were consistently 1.0 with variance 0.0, although structural similarity still indicated AST variation.

4.5 RQ5: GPT-4 vs. GPT-3.5

GPT-4 is slightly less nondeterministic than GPT-3.5 at temperature=1, but its nondeterminism is similar at temperature=0. The comparison uses CodeContests results and reports broader cross-dataset evidence for the zero-temperature case.

  • Under temperature=0, GPT-4’s nondeterminism is similar to GPT-3.5’s across CodeContests, APPS, and HumanEval.The comparison uses similarity metrics across three evaluation methods.
  • At temperature=1, GPT-4 shows lower test pass rate variance, lower worst-case ratios, lower OER measures, lower LCS, higher LED, and lower structural similarity under two settings.
  • GPT-4 is slightly less nondeterministic than GPT-3.5 under temperature=1.

4.6 RQ6: Influence of Prompt Engineering Strategies on the Non-determinism

Prompt strategy has little obvious effect on nondeterminism at temperature=1, whereas temperature=0 differentiates the strategies. Concise prompts are more deterministic, while Chain-of-Thought prompts show higher randomness.

  • The Concise prompt asks for short programs, while the CoT prompt requests reasoning steps before Python3 code.
  • Under temperature=1, nondeterminism differences among prompt-engineering techniques are not obvious across the three datasets.
  • Under temperature=0, Concise prompts produce more deterministic code candidates than the Base prompt.
  • Under temperature=0, Chain-of-Thought prompts produce more random code candidates than the Base prompt.In CodeContests, CoT has mean-variance 0.02, mean maximum difference 0.15, and a worst-case ratio of 1.82%.

5 Threats to Validity

The study identifies internal, external, and measurement-related threats to validity, alongside limitations affecting generalizability. These include implementation and analysis, selected datasets and models, similarity operationalization, programming languages, prompts, and model coverage.

  • Internal validity: Internal validity is threatened by experiment implementation and result analysis, mitigated through code checks and independent author analyses.
  • External validity: External validity is constrained by the selected datasets, GPT versions, and prompt designs, despite using diverse benchmarks and two GPT versions.
  • Measurement validity: Measuring semantic similarity through test execution outputs may oversimplify code meaning, logic, and diverse correct solutions.The authors use multiple similarity types and measurements and validate HumanEval measurements on EvalPlus.
  • Study limitations: The analysis does not cover programming-language effects, and no unified standard exists for measuring code similarity.
  • Study limitations: Prompt influence is not fully considered, and findings from the ChatGPT-only study may not apply to other LLMs.

6 Related Work

The section reviews code-generation approaches spanning deductive and inductive synthesis, neural architectures, transformers, and pre-trained models. It then motivates this study by showing that non-determinism remains insufficiently examined in LLM-based coding research.

  • Code Generation: Code-generation systems must produce programs satisfying constraints represented by input/output pairs, examples, descriptions, partial programs, or assertions.
  • Neural Code Generation: Neural code-generation research includes syntax-driven architectures, grammar-based structural CNNs, and dual learning frameworks.
  • Transformer-Based Models: Transformer-based code-generation models are classified as sequence-based, tree-based, or pre-trained approaches.
  • Model Categories: Sequence-based methods generate token sequences, whereas tree-based methods construct parse trees such as abstract syntax trees before producing code.
  • Pre-trained Models: Pre-trained code models learn from massive source-code datasets and may be fine-tuned for code-generation tasks.
  • Research Gap: ChatGPT is selected because it is widely adopted and represents state-of-the-art code-generation performance, while the severity of non-determinism in coding studies remains unclear.
  • Research Practice: Among 76 surveyed papers, 35.5% (27/76) mention nondeterminism or related terms.

7 Discussion

The discussion weighs non-determinism’s reliability costs against its potential to increase creativity and candidate diversity. It recommends accounting for variance in development and research while exploring mechanisms that balance determinism with creativity.

  • Implications for Developers: Developers should recognize ChatGPT’s limitations and production risks, using lower temperature for greater stability without assuming temperature=0 guarantees determinism.
  • Implications for Developers: Prompt length is correlated with code correctness and non-determinism, highlighting prompt engineering as an important development consideration.
  • Implications for Researchers: Researchers should account for generated-code variance because instability can produce non-reproducible results and unreliable conclusions.
  • Trade-off of Non-determinism: Non-determinism introduces consistency and reliability challenges but can also provide creativity and diversity in generated outputs.
  • Trade-off of Non-determinism: Five requests at temperature 1 improved the highest-pass-rate candidate over the lowest by 16.13 times for CodeContests, 3.12 times for APPS, and 1.98 times for HumanEval.
  • Future Directions: Proposed directions include voting mechanisms, LLM-driven repair loops, hybrid models, and user-customizable determinism levels.

8 Conclusion

The conclusion reports substantial variation among ChatGPT code candidates generated from identical instructions across three benchmarks. It calls for greater awareness of non-determinism in future LLM-based code-generation research.

  • Conclusion: Experiments on three widely studied benchmarks found significant variation in correctness, test outputs, syntax, and structure across requests with the same instruction.
Loading 2308.02828v2…