PaperlayerView exact revision on arXiv

Source-linked AI summary

ChatGPT: A Study on its Utility for Ubiquitous Software Engineering Tasks

Giriprasad Sridhara, Ranjani H. G., Sourav Mazumdar

arXiv:2305.16837v1cs.SEcs.AIcs.LG

TL;DR

The paper asks whether ChatGPT can assist with common software engineering tasks. It evaluates fifteen tasks against human or state-of-the-art outputs and finds credible performance for many tasks, while noting incorrect answers for others. The study therefore supports using ChatGPT directly or as an adjunct for some tasks, within a manually evaluated and limited-sample scope.

  • Problem

    The study addresses limited evidence about ChatGPT's usefulness for common software engineering tasks across development, quality assurance, and maintenance.

    Source: §I · p. 2

  • Method

    The authors evaluate ChatGPT on fifteen software engineering tasks and compare its outputs with human ground truth, state-of-the-art tools, or manual evaluation.

    Source: §I · p. 2

  • Results

    ChatGPT performs very well on some tasks, averages on others, and provides incorrect answers for a few tasks, making it suitable only for selected uses.

    Source: K. Code Generation from Natural… · p. 7, S. Overall Summary · p. 10, §IV · p. 11

  • Takeaways & Limitations

    ChatGPT can be used as-is for some software engineering tasks or as an adjunct tool for others.

    Source: §IV · p. 11

    The study uses small samples because the manual, time-consuming design limited the number of ChatGPT conversations.

    Source: T. Threats to Validity · p. 10

Abstract

from arXiv · show

ChatGPT (Chat Generative Pre-trained Transformer) is a chatbot launched by OpenAI on November 30, 2022. OpenAI's GPT-3 family of large language models serve as the foundation for ChatGPT. ChatGPT is fine-tuned with both supervised and reinforcement learning techniques and has received widespread attention for its articulate responses across diverse domains of knowledge. In this study, we explore how ChatGPT can be used to help with common software engineering tasks. Many of the ubiquitous tasks covering the breadth of software engineering such as ambiguity resolution in software requirements, method name suggestion, test case prioritization, code review, log summarization can potentially be performed using ChatGPT. In this study, we explore fifteen common software engineering tasks using ChatGPT. We juxtapose and analyze ChatGPT's answers with the respective state of the art outputs (where available) and/or human expert ground truth. Our experiments suggest that for many tasks, ChatGPT does perform credibly and the response from it is detailed and often better than the human expert output or the state of the art output. However, for a few other tasks, ChatGPT in its present form provides incorrect answers and hence is not suited for such tasks.

I. INTRODUCTION

The paper investigates whether ChatGPT can support fifteen common software engineering tasks by comparing its outputs with human experts and state-of-the-art tools. It reports a study spanning software development, quality assurance, and maintenance tasks.

I. INTRODUCTION

  • The research question is whether ChatGPT is useful for common software engineering tasks amid reported successes and failures across other domains.Source: §I · p. 1
  • The study evaluates ChatGPT on fifteen ubiquitous software engineering tasks spanning software development, quality assurance, and maintenance.Source: §I · pp. 1–2
  • ChatGPT outputs are compared with human expert ground truth and, where available, state-of-the-art tool outputs.Source: §I · pp. 1–2

II. STUDY

  • The study repeats experiments across randomly chosen samples and records ChatGPT accuracy for tasks including code summarization, duplicate bug detection, and test case prioritization.Source: §I · pp. 1–2

A. ChatGPT

  • ChatGPT is described as a large-scale OpenAI language model based on the GPT architecture and capable of generating human-like responses across diverse topics.Source: A. ChatGPT · p. 2

B. Study Setup

The study uses small, manually collected task samples and compares ChatGPT with human or state-of-the-art outputs where available. For method-name suggestion, ChatGPT performs strongly, producing accurate and often more precise names.

B. Study Setup

  • Each task uses at most ten samples because the manual conversations and comparison with state-of-the-art tools are time-consuming.Source: B. Study Setup · p. 2
  • The evaluation defines accuracy as the number of ChatGPT outputs matching human, state-of-the-art, or manually evaluated expected outputs.Source: B. Study Setup · p. 2

C. Method Name Suggestion or Code Summarization

D. Log Summarization

ChatGPT performed strongly on log summarization and commit message generation, but commit messages sometimes included extraneous content requiring review.

D. Log Summarization

  • ChatGPT generated appropriate summaries for all ten logs and better summaries than the state of the art in every case.Its summaries could be more informative without being overly verbose.Source: D. Log Summarization · p. 3

E. Commit Message Generation

F. Duplicate Bug Report Detection

ChatGPT identified six of ten duplicate bug-report pairs and resolved six of ten merge conflicts, indicating mixed performance across these maintenance tasks.

F. Duplicate Bug Report Detection

G. Merge Conflict Resolution

H. Anaphora Resolution

ChatGPT resolved all ten sampled anaphora cases but matched developer code reviews in only four of ten cases, with discrepancies not explained solely by missing domain knowledge.

H. Anaphora Resolution

I. Code Review

  • ChatGPT’s code-review output agreed with the original developer review in only four of ten cases.Source: §I · p. 5
  • ChatGPT exactly matched a developer’s review when it flagged misuse of Java’s AccessController and recommended the System Property alternative.Source: §I · p. 5
  • The developers’ project-specific domain knowledge may not have been reproducible by ChatGPT in some cases.Source: §I · pp. 5–6
  • Review discrepancies could not be attributed solely to domain knowledge, since ChatGPT missed a severe error logged at the lowest level.Source: §I · p. 6
  • The study concludes that ChatGPT in its current form cannot be used for code-review generation.Source: §I · p. 6

J. Type Inference Suggestion

ChatGPT showed useful but imperfect performance for type inference and natural-language dataframe code generation. It succeeded on most sampled cases while making errors involving library-specific types, unavailable function bodies, feature selection, and test prioritization.

J. Type Inference Suggestion

  • ChatGPT inferred variable types correctly in seven of ten Python functions and was correct in a non-obvious mixed-type return case.Errors involved confusing similarly named array types across libraries and inferring a return type without access to a called function’s body.Source: J. Type Inference Suggestion · p. 6

K. Code Generation from Natural Language for Data Frame Analysis

  • ChatGPT generated functioning dataframe-analysis code in eight of ten cases, including suitable pandas code with meaningful identifiers and comments.The evaluation copied generated code into a Jupyter notebook and executed it.Source: K. Code Generation from Natural… · p. 6
  • The generated dataframe code correctly computed execution-time differences and interpreted zero as a failed test run in this dataset.These behaviors required understanding the meanings of the end-time, start-time, and result columns.Source: K. Code Generation from Natural… · pp. 6–7
  • ChatGPT initially failed to select test-case name and execution time as classifier features, requiring a follow-up prompt to identify them.The authors characterize this as a weakness in feature engineering for distinguishing positive and negative samples.Source: K. Code Generation from Natural… · p. 7
  • ChatGPT produced an incorrect test-prioritization order by reversing the meaning of result value 1, despite correctly counting zeros as faults in an earlier question.This inconsistency illustrates that suitable natural-language understanding did not guarantee correct task-specific code.Source: K. Code Generation from Natural… · p. 7

L. Defect or Vulnerability Detection

ChatGPT performed poorly on vulnerability detection in the sampled methods. The authors attribute the difficulty to the extensive control- and data-flow analysis required and conclude that it should not be used as-is for this task.

L. Defect or Vulnerability Detection

  • ChatGPT correctly identified vulnerabilities in only four of ten sampled methods.The study asked ChatGPT to classify methods drawn from a vulnerability dataset.Source: L. Defect or Vulnerability Detection · p. 7
  • The authors suggest that vulnerability detection exceeds ChatGPT’s capabilities because it requires extensive control- and data-flow analysis, particularly for low-level C code with pointers and indirection.They conclude that ChatGPT cannot be used as-is for vulnerability detection.Source: L. Defect or Vulnerability Detection · p. 7

M. Code Clone or Duplicate Code Detection

ChatGPT showed moderate-to-strong performance on code-clone identification and test-oracle generation, while its disagreements often reflected semantic distinctions or assertion-context errors. The authors view it as useful for clone detection and as a supervised aid for generating assertions.

M. Code Clone or Duplicate Code Detection

  • ChatGPT correctly identified six of ten code-clone pairs, including clones with different names, parameters, and implementations.In three of four apparent errors, the authors judged the benchmark ground truth incorrect; the remaining case involved a semantic distinction between copying one file and recursively copying multiple files.Source: M. Code Clone or Duplicate Code… · p. 7
  • The authors conclude that ChatGPT makes fine-grained distinctions and cogent arguments in code-clone detection.Its performance was judged good despite disagreement with some benchmark labels.Source: M. Code Clone or Duplicate Code… · p. 7

N. Test Oracle Generation

  • ChatGPT’s generated assert statement matched the developer ground truth six of ten times.It also supplied a cogent explanation of the assertion.Source: N. Test Oracle Generation · p. 8
  • Failures in test-oracle generation included reversed assertions, use of a focal-method-local variable, and one case where the test targeted a vector operation rather than the focal constructor.The authors identified the latter mismatch as an anomaly in the test setup.Source: N. Test Oracle Generation · p. 8
  • The authors characterize oracle generation as average but suggest that ChatGPT can generate assert statements with manual supervision.This recommendation is narrower than treating the generated assertions as reliable without review.Source: N. Test Oracle Generation · p. 8

O. Code Refactoring

ChatGPT produced semantically valid, readable refactorings even when none matched developers’ exact ExtractMethod choices, and matched code-search ground truth in most sampled cases. These results support assisted use while preserving the subjectivity and error sensitivity of the tasks.

O. Code Refactoring

  • ChatGPT’s ExtractMethod refactorings matched the original developer refactorings in none of the samples.ExtractMethod decisions are inherently subjective, so disagreement alone does not rule out usable refactorings.Source: O. Code Refactoring · p. 8
  • Despite differing from developers’ choices, ChatGPT’s refactorings were syntactically correct, preserved computational intent, and improved readability with good identifiers.The authors found the extracted methods, parameters, and return values clearly named.Source: O. Code Refactoring · p. 8

P. Natural Language Code Search

  • ChatGPT matched code-search ground truth seven of ten times when judging whether snippets answered natural-language queries.The evaluation used query-snippet pairs from the CoSQA dataset.Source: P. Natural Language Code Search · p. 9
  • ChatGPT distinguished subtle meanings such as relative versus absolute paths, while some apparent failures reflected an erroneous ground truth or an uninformative query.The authors therefore describe code search as a task where ChatGPT could serve as an adjunct tool.Source: P. Natural Language Code Search · p. 9

Q. Test Case Prioritization

The study examines three ways to prioritize software tests and finds that ChatGPT succeeds for past-fault ordering but is not viable for code-change or operation-order prioritization.

Q. Test Case Prioritization

  • The study explored prioritization by past faults, code changes, and operation order because frequent code updates make running every test impractical.The three approaches respectively use historical faults, code-diff similarity, and API operation dependencies.Source: Q. Test Case Prioritization · p. 9

1) Prioritization based on past faults found:

  • ChatGPT produced the correct CEBAD order for prioritization based on previously detected faults, although its reasoning used a naive and potentially suboptimal strategy.Executing C and E first detects all faults, but ChatGPT’s explanation favored tests finding the most faults, which can yield suboptimal orders.Source: 1) Prioritization based on past faults… · p. 9

2) Prioritization based on code changes:

  • ChatGPT could not prioritize tests using code diffs and textual similarity, despite performing related code clone and duplicate bug report tasks.This approach uses cosine similarity between code changes and test cases and is based on information retrieval.Source: 2) Prioritization based on code changes · p. 9

3) Prioritization based on operation order:

R. Other Tasks

The paper reports mixed results across additional software engineering tasks: ChatGPT failed at deobfuscation and automated repair, matched solutions for sampled efficient algorithms, and required manual study constraints.

1) Code Deobfuscation:

  • ChatGPT was unable to recover original identifiers during JavaScript deobfuscation, so it may not be useful for this task.The task matters for comprehension and security review because minification replaces meaningful names with shorter identifiers.Source: 1) Code Deobfuscation · pp. 9–10

2) Efficient Algorithm Generation::

  • ChatGPT generated code matching ground-truth solutions for sampled efficient-algorithm problems, although the study deferred evaluation on the standard ten samples.The sampled problems included time-complexity constraints such as O(n).Source: 2) Efficient Algorithm Generation · p. 10

3) Automated Program Repair::

  • ChatGPT could not repair the sampled buggy Java methods because their identifiers had been normalized, with evaluation on original buggy methods left for future work.The inputs came from CodeXGLUE and replaced identifiers with normalized forms such as TYPE 1 and VAR 1.Source: 3) Automated Program Repair · p. 10

S. Overall Summary:

  • The study summarized ChatGPT’s performance across the software engineering tasks in Table II, whose double asterisk marks tasks ChatGPT could not perform across multiple samples.Source: S. Overall Summary · p. 10, Table II · p. 10

T. Threats to Validity

  • The study used a potentially small number of samples because manually conducting the conversations was time-consuming.The authors note that the study had to limit the number of conversations for this reason.Source: T. Threats to Validity · p. 10
  • The selected tasks were intended to represent software engineering activities through industry knowledge, open-source project activities, and research trends.Source: T. Threats to Validity · p. 10

III. RELATED WORK

The related work situates ChatGPT among code-focused language models and benchmarks, while the conclusion evaluates it across fifteen software engineering tasks against expert or state-of-the-art outputs.

III. RELATED WORK

  • Prior work includes CodeBERT, CodeXGLUE, Codex, InstructGPT, open-source language models, and evaluations of code-language models.These studies cover code understanding, generation, benchmarking, instruction following, and open-source alternatives.Source: §III · pp. 10–11
  • The paper claims its study is the first rigorous evaluation of ChatGPT across different software engineering tasks.It characterizes ChatGPT as consolidating code-to-text, text-to-text, and instruction-following advances.Source: §III · p. 11

IV. CONCLUSION

  • The paper asks how useful ChatGPT is for common software engineering tasks such as code summarization and test oracle generation.Source: §IV · p. 11
  • The authors evaluated fifteen tasks using ten random samples per task and compared ChatGPT’s answers with human expert or state-of-the-art outputs.They computed ChatGPT’s accuracy for each task.Source: §IV · p. 11
  • ChatGPT performed very well on log summarization, anaphora resolution, method-name generation, and code clone detection, but only average on several other tasks.Average-performance tasks included commit message generation, code review generation, natural-language code search, and merge conflict resolution.Source: §IV · p. 11
  • ChatGPT performed poorly on vulnerability detection and could not answer information-retrieval-based test prioritization, so some tasks require careful output checking.Overall, the authors describe ChatGPT as usable directly for some tasks and as an adjunct tool for others.Source: §IV · p. 11

Related papers

Loading 2305.16837v1…