Source-linked AI summary
Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generation
Li Zhong, Zilong Wang
TL;DR
ROBUSTAPI addresses the limited study of reliability and robustness in LLM-generated code beyond functional correctness, especially for real-world software development. It benchmarks API misuse using Stack Overflow-based Java questions and static analysis, finding that API misuse widely exists even when generated code is executable and aligned with users’ intentions.
Problem
The reliability and robustness of LLM-generated code remain insufficiently studied because existing evaluations emphasize functional correctness and coding-challenge tasks rather than real-world software development risks.
Method
ROBUSTAPI combines Stack Overflow coding questions about Java APIs with an AST-based checker that evaluates API usage rules in generated code.
Results
API misuse widely exists in LLM-generated code even when the code is executable and aligned with users’ intentions.
Takeaways & Limitations
The benchmark adds API-usage reliability and robustness as an evaluation dimension for LLM code generation and supports further study of improving misuse rates.
Takeaways & Limitations
ROBUSTAPI focuses on Java API usage, although the proposed method could theoretically be applied to languages such as Python.
Abstract
from arXiv · showhide
Recently, the large language models (LLMs) have shown extraordinary ability in understanding natural language and generating programming code. It has been a common practice of software engineers to consult LLMs when encountering coding questions. Although efforts have been made to avoid syntax errors and align the code with the intended semantics, the reliability and robustness of the code generationfrom LLMs have not yet been thoroughly studied. The executable code is not equivalent to the reliable and robust code, especially in the context of real-world software development. The misuse of APIs in the generated code could lead to severe problem, such as resource leaks, program crashes. To make things worse, the users of LLM code generation services are actually the developers that are most vulnerable to these code that seems right -- They are always novice developers that are not familiar with the APIs that LLMs generate code for them. Therefore, they could hardly tell the misuse in the code generated by LLMs, which further facilitates the incorrect code applied in real-world software. Existing code evaluation benchmark and datasets focus on crafting small tasks such as programming questions in coding interviews, which however deviates from the problem that developers would ask LLM for real-world coding help. To fill the missing piece, in this work, we propose a dataset RobustAPI for evaluating the reliability and robustness of code generated by LLMs. We collect 1208 coding questions from StackOverflow on 24 representative Java APIs. We summarize thecommon misuse patterns of these APIs and evaluate them oncurrent popular LLMs. The evaluation results show that evenfor GPT-4, 62% of the generated code contains API misuses,which would cause unexpected consequences if the code isintroduced into real-world software.
Introduction
LLMs offer customized coding help, but generated code can remain unreliable and risky even when executable or functionally correct. ROBUSTAPI addresses this gap by evaluating API misuse in Stack Overflow-based software-development questions.
- Motivation: LLM-generated code can contain API misuses such as missing boundary checks, unclosed file streams, and incomplete transactions.These issues may cause memory leaks, program crashes, and other production risks.
- Motivation: Existing benchmarks mainly assess functional correctness on small coding-challenge tasks rather than reliability in practical software development.The paper argues that functional execution alone does not establish long-term code reliability.
- Contribution: ROBUSTAPI evaluates reliability and robustness using Stack Overflow coding questions, an AST-based evaluator, and formalized API usage patterns.The benchmark is designed to approximate real software-development requests.
- Contribution: The benchmark collects 1208 real Java questions and evaluates GPT-3.5, GPT-4, Llama-2, and Vicuna-1.5 under zero-shot and one-shot settings.The study also analyzes common API misuse cases across these models.
- Contribution: The paper presents API misuse as an under-studied dimension of code quality beyond functional correctness.Its stated contributions include benchmarking popular LLMs and identifying promising directions for improvement.
Related Work
Prior work has examined correctness, code smells, bugs, security, and forum-code quality, but most LLM benchmarks still emphasize functional correctness and simple programming tasks.
- LLM-generated code: Research on LLM-generated code has assessed compilation, functional correctness, efficiency, code smells, implementation errors, and difficult-to-detect bugs.These studies establish several dimensions of generated-code quality but do not resolve API-misuse evaluation.
- LLM-generated code: Approximately 40% of code generated for 89 security-sensitive Copilot scenarios was reported as vulnerable.This prior result highlights security risks in LLM-assisted code generation.
- Benchmark gaps: Existing programming benchmarks remain focused on functional correctness and simple questions that are far from software-development scenarios.The related work contrasts these benchmarks with reliability concerns addressed in this paper.
- Forum code quality: Studies of online programming forums have reported compilation failures, deprecated APIs, and security issues in forum-provided code.These findings motivate examining code quality beyond whether generated code executes.
Methodology
ROBUSTAPI constructs a Stack Overflow-based Java benchmark, prompts LLMs under controlled demonstration settings, and uses static AST analysis to detect violations of formal API usage rules.
- Data Collection: The dataset selects 18 popular Java APIs across six domains and retrieves related Stack Overflow questions with online answers.The domains include string processing, data structures, mobile development, cryptography, I/O, and database operations.
- Data Collection: ROBUSTAPI represents each sample with an identifier, API hint, Stack Overflow question text, and original URL.The collected questions are converted into JSON format for evaluation.
- Prompt Generation: Prompts combine task instructions, a required response format, optional demonstrations, and the question with its API hint.This setup simulates novice developers seeking help without additional API documentation.
- Prompt Generation: The experiments compare irrelevant-API and relevant-API one-shot demonstrations, with relevant examples manually revised to avoid API misuse.The settings test whether demonstrations affect generated-code quality and API usage.
- API Usage Rules: The RandomAccessFile example illustrates that correct file handling combines exception protection with closing the file resource.The displayed code shows a try-catch structure and an explicit raf.close() call.
- API Usage Rules: The benchmark formalizes 41 API usage rules covering guard conditions, required call sequences, and control structures.Examples include checking File.exists() before File.createNewFile(), closing after writing, and enclosing parsing in try-catch.
- Detecting API Misuse: Static analysis builds an AST, extracts method calls and control structures into call sequences, and compares them with API usage rules.The checker infers method types, retrieves matching rules, and computes sequence similarity to identify misuse.
Experimenet
ROBUSTAPI evaluates four LLMs under zero-shot and one-shot conditions, using API-misuse and compilation metrics to assess generated code.
- The evaluation covers GPT-3.5, GPT-4, Llama2, and Vicuna-1.5 using default hyper-parameter settings.
- The three prompt settings are zero-shot, one-shot-irrelevant, and one-shot-relevant.
- The manually written shot examples are double-checked and evaluated against API usage checkers for alignment with usage rules.
- Metrics: API Misuse Rate measures misuse among compilable snippets, while Compilation Rate measures compilable cases among all questions.
- Evaluation Method: The API checker analyzes generated code through abstract syntax trees and detects violations when call sequences conflict with API usage rules.
Research Questions
The experiments use ROBUSTAPI to examine API misuse, prompt-example effects, correct usage examples, and reasons generated code fails API checks.
- Research Questions: The experiments evaluate how often LLM answers to real-world coding questions contain API misuse.
- Research Questions: They test whether irrelevant examples affect results and whether correct API usage examples reduce misuse.
- Research Questions: They investigate why LLM-generated code fails the API usage check.
API Misuse Rate
ROBUSTAPI finds widespread API misuse in LLM-generated answers, including state-of-the-art models, while code-generation ability does not ensure reliable API usage.
- All evaluated models suffer from API misuse problems, including GPT-3.5 and GPT-4.
- GPT-4 has a higher API misuse rate than GPT-3.5 in zero-shot settings despite its reported coding improvements.
- Llama has the lowest zero-shot API misuse rate, partly because most of its answers do not include code.
- Code-specialized models generate more compilable samples, but their API misuse rate is not significantly better than other models.
- API misuse remains a major reliability and robustness issue, leaving substantial room for improvement.
- Static-analysis execution times vary with LLM coding styles but remain within 7 minutes.
One-Shot-Irrelevant Results
Irrelevant examples generally did not reduce API misuse, while relevant correct-usage examples reduced misuse for several models.
- Findings: 57-70% of answers containing compilable code contained API misuse.These misuses could lead to severe consequences in production.
- Irrelevant shots: Irrelevant shots generally failed to reduce API misuse and slightly increased it for most models.They encouraged longer code solutions, increasing opportunities for misuse.
- Irrelevant shots: Irrelevant shots increased the number of valid answers containing code snippets, enabling broader evaluation of reliability and robustness.
- Relevant shots: Relevant correct-usage shots significantly reduced misuse rates for GPT-3.5, GPT-4, and Vicuna, but not Llama.The result indicates that some models can learn correct API usage from examples.
Robustness Analysis
The robustness analysis finds that temperature changes and replacing relevant examples with API rules do not significantly alter misuse rates.
- Temperature: Increasing temperature did not significantly change GPT-3.5’s misuse rate or compilation rate.
- API usage rules: Replacing one-shot examples with API usage rules did not significantly affect the API misuse rate.For GPT-3.5, API rules might not reduce misuse compared with relevant one-shot examples.
Error Analysis
Error analysis separates compilation failures from compilable code containing API misuses and examines model- and API-specific patterns.
- API-specific misuse: The misuse-rate figure encodes each API–LLM combination by color depth, with darker colors indicating higher misuse.Its abbreviations include G3.5 for GPT3.5, G4 for GPT4, LMA for Llama2, and Vic for Vicuna1.5.
- Compilation failures: GPT-4 had fewer than 10% uncompilable answers across all experiment settings.Adding a few prompt shots reduced compilation failure rates for all models.
- API-specific misuse: Activity.setContentView had the lowest misuse rate across all evaluated models.
- Case study: For PrintWriter.write, zero-shot and irrelevant-shot outputs omitted exception handling, whereas the relevant example produced correct usage.
Discussion
ROBUSTAPI focuses on Java API misuse while identifying broader directions for evaluating and improving LLM-generated code in real-world development.
- Scope: ROBUSTAPI focuses on Java API usage because Java is widely used and important in web and Android ecosystems.The authors state that the method could theoretically extend to languages such as Python.
- Future work: Future work includes improving generated-code quality beyond functionality alignment through in-context learning, fine-tuning, and pre-training.
- Future work: The authors identify real-world software-development evaluation and improvement of LLMs as an important continuing research problem.
Conclusion
ROBUSTAPI shows that API misuse is widespread in LLM-generated code, even when the code is executable and matches users’ intentions.
- API misuse widely exists in state-of-the-art LLM-generated code despite executability and alignment with users’ intentions.The benchmark studies this behavior across different experimental settings.