PaperlayerView exact revision on arXiv

Source-linked AI summary

From Misuse to Mastery: Enhancing Code Generation with Knowledge-Driven AI Chaining

Xiaoxue Ren, Xinyuan Ye, Dehai Zhao, Zhenchang Xing, Xiaohu Yang

arXiv:2309.15606v1cs.SE

TL;DR

LLMs can generate code, but reliable exception handling remains a challenge. KPC chains fine-grained API knowledge prompts with iterative checking and rewriting; evaluations report improvements over ChatGPT across static and dynamic validation.

  • Problem

    LLM-generated code can have incomplete or incorrect exception handling and inappropriate try-catch use, threatening code reliability.

    Source: §I · pp. 1–2

  • Method

    KPC uses fine-grained exception-handling knowledge from official API documentation in chained prompts that iteratively check and rewrite generated code.

    Source: §I · p. 2

  • Results

    109.86% and 578.57% improvements in the two static evaluations, and 18 fewer runtime bugs, were reported for KPC compared with ChatGPT.

    Source: §I · p. 2

  • Takeaways & Limitations

    KPC can remind developers of exception-handling specifications, with the user study reporting 75.00% correctness in handling potential exceptions.

    Source: §I · p. 2

    The authors acknowledge that KPC has limitations.

    Source: A. Limitations of KPC · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) have shown promising results in automatic code generation by improving coding efficiency to a certain extent. However, generating high-quality and reliable code remains a formidable task because of LLMs' lack of good programming practice, especially in exception handling. In this paper, we first conduct an empirical study and summarise three crucial challenges of LLMs in exception handling, i.e., incomplete exception handling, incorrect exception handling and abuse of try-catch. We then try prompts with different granularities to address such challenges, finding fine-grained knowledge-driven prompts works best. Based on our empirical study, we propose a novel Knowledge-driven Prompt Chaining-based code generation approach, name KPC, which decomposes code generation into an AI chain with iterative check-rewrite steps and chains fine-grained knowledge-driven prompts to assist LLMs in considering exception-handling specifications. We evaluate our KPC-based approach with 3,079 code generation tasks extracted from the Java official API documentation. Extensive experimental results demonstrate that the KPC-based approach has considerable potential to ameliorate the quality of code generated by LLMs. It achieves this through proficiently managing exceptions and obtaining remarkable enhancements of 109.86% and 578.57% with static evaluation methods, as well as a reduction of 18 runtime bugs in the sampled dataset with dynamic validation.

I. INTRODUCTION

LLMs can generate code, but exception handling remains a challenge for reliable, maintainable, and robust code. KPC chains fine-grained API knowledge prompts with iterative checks and rewrites; evaluations report improvements over ChatGPT.

I. INTRODUCTION

  • LLMs face incomplete and incorrect exception handling and abuse of try-catch, which can contribute to crashes and reliability and security issues.The paper identifies all three as challenges in its empirical study.Source: §I · pp. 1–2
  • Fine-grained knowledge-driven prompts were most effective among the prompt granularities tried for preventing incomplete or incorrect handling and try-catch abuse.The approach was evaluated on 3,079 Java coding tasks drawn from official API documentation.Source: §I · p. 2
  • KPC uses fine-grained exception-handling knowledge from API documentation in chained prompts that check and rewrite generated code until exceptions are handled.The approach constructs an API knowledge base and applies the prompts iteratively to code initially generated by an LLM.Source: §I · p. 2
  • 109.86% and 578.57% improvements in the two static evaluations, plus 18 fewer runtime bugs, were reported for KPC versus ChatGPT.A user study also found 75.00% correctness in handling potential exceptions.Source: §I · p. 2

II. AN EMPIRICAL STUDY ON EXCEPTION HANDING FOR LLMS CODE GENERATION

The empirical study examines exception-handling challenges in LLM-generated Java code and ways prompts might address them. It identifies incomplete and incorrect exception handling among the challenges observed in a manually reviewed sample.

II. AN EMPIRICAL STUDY ON EXCEPTION HANDING FOR LLMS CODE GENERATION

  • The study summarizes three challenges: incomplete handling, incorrect handling, and abuse of try-catch statements.The two research questions address the challenges and how prompts can help address them.Source: §II · p. 2

A. Challenges of LLMs in Exception Handling (RQ1)

  • 92 Java coding tasks with potential exceptions were generated by ChatGPT and manually reviewed by two authors experienced in Java development and API documentation.The tasks were collected from Tutorialspoint and rephrased as inputs to ChatGPT.Source: A. Challenges of LLMs in Exception… · p. 3

1) Incomplete exception handling:

  • 88.04% (81 out of 92) of the ChatGPT-generated tasks had incomplete exception handling, despite relevant API specifications being available in official documentation.Uncaught exceptions can lead to crashes, data corruption, and security vulnerabilities.Source: 1) Incomplete exception handling · p. 3

2) Incorrect exception handling:

  • 4.35% (4 out of 92) of ChatGPT-generated examples contained incorrect exception handling, such as catching a broader exception type than the API recommends.The paper notes that such examples may run without error, making the root cause difficult to locate.Source: 2) Incorrect exception handling · p. 3

3) Abuse of try-catch:

The study describes try-catch abuse and compares prompting strategies for exception handling. General prompts reduce incomplete handling but increase other issues; coarse-grained prompts reduce incomplete and incorrect handling while increasing try-catch abuse.

3) Abuse of try-catch:

  • ChatGPT used try-catch in 5.43% (5 out of 92) of tasks; the study identifies incomplete handling, incorrect handling, and try-catch abuse as key challenges.The authors note that inappropriate try-catch use can reduce code readability and maintainability.Source: 3) Abuse of try-catch · p. 3

B. Prompt Engineering for Exception Handling (RQ2)

  • The prompt study progressively adds exception-handling information at general, coarse-grained, and fine-grained levels.The strategy was designed to examine how prompt information affects exception handling in code generation.Source: B. Prompt Engineering for Exception… · p. 4

1) General prompting:

  • General prompting reduced incomplete-handling examples from 81 to 35, while incorrect handling rose from 4 to 23 and try-catch abuse from 5 to 20.The prompt used a generic warning but did not supply exception types or conditions.Source: 1) General prompting · p. 4

2) Coarse-grained prompting:

  • 95.06% fewer incomplete-handling cases (from 81 to 4) and no incorrect-handling cases were observed with coarse-grained prompting, but try-catch abuse rose from 5 to 41.The remaining incomplete cases involved APIs with incomplete exception specifications in the official documentation.Source: 2) Coarse-grained prompting · p. 4

3) Fine-grained prompting:

Fine-grained prompts supply exception types and triggering conditions, addressing exception-handling mistakes more effectively than less detailed prompts. KPC builds on this knowledge-driven prompting in a chained code-generation approach.

3) Fine-grained prompting:

  • Fine-grained prompts completely eliminate incorrect exception handling and try-catch abuse, while three examples still have incomplete handling.The remaining cases are attributed to uncertainty in ChatGPT's outputs.Source: 3) Fine-grained prompting · p. 4, Table I · p. 5
  • The final prompting strategy supplies both exception types and their conditions, such as an index-range check before throwing ArrayIndexOutOfBoundsException.Source: 3) Fine-grained prompting · p. 4
  • Specific exceptions and their triggering conditions enable LLMs to identify and handle exceptions accurately, following software-development best practice.Source: 3) Fine-grained prompting · p. 4

III. THE APPROACH

  • The framework separates API knowledge-base construction into an offline phase and KPC-based code generation into an online phase.Source: §III · p. 4, Figure 3 · p. 5
  • The study states that its three exception-handling challenges do not overlap and prioritizes incomplete handling, incorrect handling, then try-catch abuse.Source: §III · p. 5
  • KPC applies knowledge-driven prompt chaining on top of ChatGPT to improve exception-handling quality.Source: §III · p. 5

A. API Knowledge Base Construction

1) Knowledge schema:

The method represents API exception knowledge as relations among APIs, exceptions, and triggering conditions, then uses an LLM chain to generate and inspect code. Checking identifies APIs and asks whether their associated exceptions are handled.

1) Knowledge schema:

  • The knowledge schema links APIs, exceptions, and conditions through two relations: APIs throw exceptions, and conditions trigger exceptions.Source: 1) Knowledge schema · p. 5

2) Knowledge extraction:

  • Knowledge extraction keeps API declarations and exception-handling specifications from Java SDK and JDK documentation, using rules to extract exceptions and conditions.The work focuses on method-level APIs and does not further preprocess condition descriptions.Source: 2) Knowledge extraction · p. 5

B. KPC-based Code Generation

1) Generating:

  • A natural-language task is reframed as a request to write a Java method, and ChatGPT generates the initial code.Source: 1) Generating · p. 5

2) Checking:

  • Checking extracts fully qualified API names from generated code, links them to the knowledge base, and collects whether each API's associated exceptions are handled.Source: 2) Checking · p. 6

3) Rewriting:

When checking finds unhandled exceptions, KPC prompts the LLM with the relevant API, condition, and exception, then repeats checking and rewriting. Evaluation compares KPC with ChatGPT on Java API tasks involving potential exceptions.

3) Rewriting:

  • When unhandled exceptions remain, rewriting prompts the LLM with each API's condition and exception, then checking and rewriting repeat until none remain.Source: 3) Rewriting · p. 6

A. Research Question

  • RQ3 measures potential exceptions and the checking-rewriting loops needed to address them; RQ4 compares KPC with ChatGPT using static and dynamic evaluation.Source: A. Research Question · p. 6

B. Task Collection and Implementation

V. EVALUATION

Most KPC tasks finish in a small number of checking-rewriting loops, with 90.35% completed within ten. The average number of unhandled exceptions can fluctuate between loops, but none remain at completion.

1) Method:

  • Efficiency is assessed by counting exceptions addressed in each loop and the number of loops required; the initial checking-rewriting round counts as loop one.Source: 1) Method · p. 6

2) Result:

  • 90.35% (2,782) of 3,079 tasks finish within 10 loops, and 58.46% (1,800) need only two loops.The reported range is 2 to 28 loops.Source: 2) Result · pp. 6–7, Figure 4 · p. 7
  • The average unhandled exceptions per loop ranges from 1.61 to 3.35 across tasks completed in the same number of loops.Source: 2) Result · p. 7
  • Average unhandled exceptions can fluctuate between loops because rewriting may introduce new ones, but all are resolved by completion.Source: 2) Result · p. 7

1) Method:

The evaluation combines automated, manual, and test-based checks of exception handling. KPC outperforms ChatGPT across these methods, with specific exception knowledge offered as an explanation for the gap between automated and manual review.

1) Method:

  • LLMEva, CodeReview, and EvoSuite assess generated code through automated static analysis, manual review, and generated test cases, respectively.LLMEva covers all tasks; CodeReview and EvoSuite use a random sample of 384 tasks, and EvoSuite uses branch and exception coverage objectives.Source: 1) Method · p. 7

2) Result:

  • Across all three evaluation methods, KPC showed exception-handling improvements of 107.86% and 578.57%, and reduced real bugs by 18 compared with ChatGPT-generated code.In EvoSuite, 31 unique bugs were detected in ChatGPT-generated code and 13 in KPC-generated code across 384 randomly selected tasks over 10 runs.Source: 2) Result · pp. 7–8
  • LLMEva rated 71.29% of KPC-generated code and 34.30% of ChatGPT-generated code as following good exception-handling practice, though LLMEva’s reliability is uncertain.The authors attribute the gap between LLMEva and manual review to LLMs’ limited exception-handling knowledge and preference for general exceptions over specific ones.Source: 2) Result · pp. 7–8
  • 98.86% of KPC-generated code met good exception-handling practice in manual review, compared with 14.58% of ChatGPT-generated code.The two reviewers’ Cohen’s Kappa was 0.98.Source: 2) Result · pp. 7–8, Table II · p. 7

VI. USER STUDY

The user study tests whether reference code from KPC helps developers complete Java coding tasks efficiently and correctly. Twelve developers completed six tasks under three conditions, with correctness, time, and perceived usefulness assessed.

1) Tasks and Procedure:

  • The study used six Java API tasks spanning easy, medium, and hard difficulty, with ten minutes allowed per task and online resources permitted except code generators.Participants also completed a short survey after the tasks.Source: 1) Tasks and Procedure · p. 8

2) Participants:

  • Twelve developers with 1–5 years of Java experience were divided into three groups: no reference code, ChatGPT reference code, or KPC reference code.Each group contained four participants.Source: 2) Participants · p. 8

3) Evaluation Metrics:

  • Performance was assessed by task completion time and answer correctness, with two authors independently reviewing submitted code and resolving disagreements.A supplementary 0–4 rating asked participants using reference code how helpful it was.Source: 3) Evaluation Metrics · p. 8

B. Result

The study found higher correctness when participants had reference code, particularly KPC-generated code, while reference-code groups took longer. Participants considered both sources helpful, and interviews highlighted difficulty identifying exception-handling flaws.

1) Correctness:

  • 75.00% of the KPC-reference group completed tasks correctly on average, versus 45.83% with ChatGPT reference code and 29.17% without reference code.Differences between all three groups were statistically significant at p-value < 0.05.Source: 1) Correctness · pp. 8–9, Table III · p. 9
  • KPC-reference participants correctly completed tasks T1, T2, and T5, while missing exception specifications in API documentation contributed to incorrect T6 reference code.ChatGPT reference code yielded no improvement for half the tasks, including T2, T3, and T6.Source: 1) Correctness · p. 9

2) Time Consumption:

  • KPC-reference participants averaged 246.3 seconds per task, compared with 242.3 seconds for ChatGPT-reference participants and 195.7 seconds without reference code.All pairwise differences in task completion time were statistically significant at p-value < 0.05; reference-code users spent about 25% more time reading it.Source: 2) Time Consumption · p. 9

3) Usefulness and Interview:

  • Participants rated ChatGPT and KPC reference code similarly helpful, at 3.5 and 3.6, while interviews found developers struggled to identify exception-handling flaws.Interviewees also noted that flawed ChatGPT code can appear correct and mislead developers.Source: 3) Usefulness and Interview · p. 9

VII. DISCUSSION

KPC’s performance is bounded by ChatGPT’s code-understanding limits and the completeness of official API exception specifications. More iterations can address exceptions but may make generated code harder to understand, creating an efficiency trade-off.

1) Code understanding:

  • ChatGPT can recognize basic syntax but lacks deep understanding of programming concepts and semantics, limiting comprehensive solutions to complex tasks.The authors identify this code-understanding limitation as a potential negative impact on KPC, which uses prompts in most steps.Source: 1) Code understanding · pp. 9–10

2) Trade-off between efficiency and effectiveness:

  • More KPC iterations increase the probability of resolving all exceptions but can make code more complex and harder for developers to understand.The authors suggest keeping the loop count within 10 to balance efficiency and effectiveness.Source: 2) Trade-off between efficiency and… · p. 10

1) Internal Validity:

  • Labels of generated-code exception handling and scores of user answers may be inaccurate, although two experienced Java annotators independently checked information about vulnerabilities.The annotators discussed cases when they could not agree.Source: 1) Internal Validity · p. 10

2) External Validity:

  • Official API documentation may omit useful exception-handling solutions, which KPC consequently cannot incorporate.The authors identify this as an external-validity threat and propose extending the knowledge base with other sources.Source: 2) External Validity · p. 10

VIII. RELATED WORK

Prior work spans LLM prompting, code generation, and automated exception handling. The conclusion presents KPC as the first knowledge-driven prompt-chaining approach, using official API specifications to guide exception handling while recognizing limits in code analysis and documentation availability.

A. Large Language Model & Prompt Engineering

  • Prompt research includes recursive plan-write-revision methods, search-guided test generation, and retrieval-based prompt selection for test assertions and program repair.The cited work notes that LLM robustness and performance depend strongly on prompt quality.Source: A. Large Language Model & Prompt… · p. 10

B. Code Generation

  • Code-generation work includes models such as Codex and Copilot, while this paper focuses on exception handling to make generated code samples more reliable.Prior work also explored programmer interaction with Copilot and LLM-generated code samples and tests.Source: B. Code Generation · p. 10

C. Exception Handling

  • Earlier exception-handling work studied links to software quality, predicted exception types, generated catch blocks, and applied LLMs to error explanations and fixes.This paper combines LLM few-shot learning with API-document knowledge to guide exception-code generation.Source: C. Exception Handling · p. 10

IX. CONCLUSION

  • KPC is presented as the first knowledge-driven prompt-chaining code-generation approach, using API specifications to build a knowledge base and fine-grained prompt chains for exception handling.The authors report that KPC is efficient and effective and that its usefulness was demonstrated in practice.Source: §IX · pp. 10–11
Loading 2309.15606v1…