Source-linked AI summary
A study on Prompt Design, Advantages and Limitations of ChatGPT for Deep Learning Program Repair
Jialun Cao, Meiziniu Li, Ming Wen, Shing-chi Cheung
TL;DR
DL program repair is challenging because intended behavior is not directly represented in source code, and the effectiveness of ChatGPT for this task is an open question. The paper studies debugging, prompting, and dialogue, reporting both useful repair capabilities and limitations.
Problem
Whether ChatGPT can effectively repair DL programs remains an open question because their decision logic is not directly encoded in source code and repair requires understanding code intention.
Source: §1 · p. 1Method
The study evaluates ChatGPT’s DL debugging, analyzes developer queries to design an enhanced prompt template, and explores repair through dialogue.
Source: §1 · pp. 1–2, §5 · p. 6Results
ChatGPT handles API misuse and deprecation well but is weaker at default parameter identification; code intention can improve repairs, while longer dialogues may bring forgetting or misunderstanding.
Source: §1 · p. 2Takeaways & Limitations
Providing more code intention can improve repairs, and dialogue may help but can also impede further improvement as input grows.
Source: §1 · p. 2ChatGPT’s library-documentation knowledge extends only to September 2021, so repairs may be invalid after newer versions are released.
Source: §6.2.1 · p. 8
Abstract
from arXiv · showhide
ChatGPT has revolutionized many research and industrial fields. ChatGPT has shown great potential in software engineering to boost various traditional tasks such as program repair, code understanding, and code generation. However, whether automatic program repair (APR) applies to deep learning (DL) programs is still unknown. DL programs, whose decision logic is not explicitly encoded in the source code, have posed unique challenges to APR. While to repair DL programs, an APR approach needs to not only parse the source code syntactically but also needs to understand the code intention. With the best prior work, the performance of fault localization is still far less than satisfactory (only about 30\%). Therefore, in this paper, we explore ChatGPT's capability for DL program repair by asking three research questions. (1) Can ChatGPT debug DL programs effectively? (2) How can ChatGPT's repair performance be improved by prompting? (3) In which way can dialogue help facilitate the repair? On top of that, we categorize the common aspects useful for prompt design for DL program repair. Also, we propose various prompt templates to facilitate the performance and summarize the advantages and disadvantages of ChatGPT's abilities such as detecting bad code smell, code refactoring, and detecting API misuse/deprecation.
1 INTRODUCTION
The introduction studies ChatGPT’s ability to debug deep-learning programs, where intended behavior is not directly encoded in source code. It frames evaluation around debugging, prompt design, and dialogue, and summarizes both repair strengths and failure modes.
1 INTRODUCTION
- Prior work reports LLM gains in program understanding and repair [48], while a 40-program study found competitive repair performance and potential prompting gains [40].Those earlier programs implement fundamental functionalities, whereas this study targets more complex DL programs.Source: §1 · p. 1
- DL programs require understanding code intention because their decision logic is not directly encoded in source code.The study uses subjects with more dependencies and functionalities than prior program-repair studies.Source: §1 · p. 1
- The study evaluates fault detection, fault localization, and program repair against two state-of-the-art techniques.These three steps are used to assess ChatGPT’s debugging performance.Source: §1 · p. 1
- The authors categorize developer queries and propose an enhanced prompt template, then evaluate it on the same benchmark.The introduction identifies what information prompts should include as an open question.Source: §1 · p. 1
- The study explores whether dialogue hints about fault locations and incorrect repairs can improve repair performance.How to conduct effective multiround dialogue for DL program repair remains unknown.Source: §1 · p. 2
- ChatGPT can handle API misuse and deprecation, but is weaker at identifying default parameters and may refactor code inconsistently.The authors also report that more code intention can improve repairs, while longer dialogue sequences may lead to forgetting or misunderstanding.Source: §1 · p. 2
2 STUDY DESIGN
The study uses StackOverflow DL programs to evaluate ChatGPT’s debugging ability against established baselines. It repeats API requests and applies defined rules for scoring detection, localization, and repair.
2.1 Benchmark
- The benchmark contains 58 buggy DL programs from StackOverflow and GitHub, but evaluation uses StackOverflow programs because GitHub programs exceed the input-token limit.Programs can contain up to five fault types; the benchmark also covers multiple DL functionalities and averages about 46 lines with more than six dependencies per program.Source: §2.1 · p. 2
2.2 Baselines
- AutoTrainer detects and repairs training problems with predefined strategies, while DeepFD classifies training faults and supports only detection and localization comparisons.DeepFD does not support program repair.Source: §2.2 · p. 2
2.3 ChatGPT Setup
- The experiments use the gpt-3.5-turbo API and run each query five times in new conversations because ChatGPT is nondeterministic.Prompts and replies are recorded and made publicly available.Source: §2.3 · p. 2
2.4 Evaluation Metrics
- Detection counts as correct when an approach flags a buggy program, while localization must match a benchmark fault and repair must fix the fault in the benchmark’s way.For epoch faults, changing the value in the correct direction counts as a correct repair even if it differs from the benchmark value.Source: §2.4 · p. 2
3 RQ1. DL PROGRAM DEBUGGING
RQ1 evaluates ChatGPT’s DL debugging under a basic prompt across fault detection, fault localization, and program repair. ChatGPT repairs some faults, but its detection and localization results trail the reported baselines.
3 RQ1. DL PROGRAM DEBUGGING
- The RQ1 baseline prompt asks whether the supplied program is buggy and, if so, requests a repair.The study applies this prompt to each buggy DL program in the benchmark.Source: §3 · pp. 2–3
3.1 Fault Detection
- DeepFD detected 34/34 buggy programs, while ChatGPT PT-B detected 27/34.ChatGPT sometimes asked for more information about code context and intention.Source: §3.1 · p. 3
- ChatGPT detected syntactic faults, functional faults, and code smells, with functional-fault detection depending on understanding code intention.Code smells may be quality or performance issues without necessarily causing functional errors.Source: §3.1 · p. 3
3.2 Fault Localization
- 29/72 faults were localized by the best existing work, compared with 23/72 by ChatGPT PT-B.The authors describe both results as far less than satisfactory.Source: §3.2 · p. 3, Table 2 · p. 4
- Without an explicit repair goal, ChatGPT tended to localize outdated APIs and code smells instead of more serious faults.The authors note that missing data preprocessing and improper training strategies could remain unlocalized.Source: §3.2 · pp. 3–4
- Misunderstanding the learning task can produce false or missed localizations, and the authors identify task knowledge and repair objectives as useful prompt information.One false localization arose when ChatGPT mistook a classification program for regression.Source: §3.2 · pp. 4–5
3.3 Program Repair
- 16/72 faults were correctly repaired by ChatGPT PT-B, twice the baseline’s 7/72; it completely repaired six cases versus AutoTrainer’s two.The overall comparison is reported in Table 2.Source: Table 2 · p. 4, §3.3 · p. 5
- Repair replies ranged from complete code and code snippets to strategy-only explanations, which can require manual fixes and trial runs.The first two forms can be checked by execution with no or little effort; variable renaming in snippets can cause crashes when lines are replaced directly.Source: §3.3 · p. 5
4 RQ2. IMPACT OF PROMPT
The study derives prompt content from real developer questions and tests an enhanced template that specifies code purpose, dataset, and repair objective. The enhanced prompt improves detection, localization, and repair, while shifting repairs toward functional faults.
4.1 Study of Real-world Questions
- The authors manually categorize text from real developer questions, excluding code blocks, to determine what information prompts commonly contain.They use StackOverflow queries for this analysis.Source: §4.1 · p. 5
- Symptom descriptions clarify the failure, task descriptions establish design purpose, and dataset details can inform preprocessing and loss-function choices.Dataset information may include data type, range, dimensions, distribution, or label count.Source: §4.1 · pp. 5–6
- Version or dependency details can guide API recommendations; attempted fixes can rule out ineffective strategies, while architecture and hyperparameter descriptions are rare.The authors note that architecture and hyperparameters may be readily identifiable from code.Source: §4.1 · p. 6
- Symptoms, tasks, and datasets are the most common of seven query-information types; they appear in 33/34, 26/34, and 24/34 descriptions, respectively.The remaining information types occur only occasionally.Source: §4.1 · p. 6, Figure 1 · p. 6
4.2 Enhanced Prompt Template Design
- PT-E asks for the learning task, dataset, and repair objective alongside the code, with variants for cases where one of those details is unavailable.The template’s X, Y, and Z fields represent task, dataset, and objective, respectively.Source: §4.2 · p. 6
4.3 Effectiveness of the Enhanced Prompt
- With PT-E, correctly repaired faults triple from 16 to 43, and fault localization doubles from 23 to 50; detection reaches 34/34 versus 27/34 with PT-B.Complete repairs increase from 6/34 to 16/34 cases.Source: §4.3 · p. 6
- Providing the repair objective reduces repairs of code smells and extra functionality while increasing true positives, shifting attention toward functionality faults.The authors attribute the shift to clearer information about code purpose, symptoms, and dataset characteristics.Source: §4.3 · p. 6
5 RQ3. EFFECTIVENESS OF CHATGPT’S DIALOGUE
The dialogue study adds fault-location hints over successive repair rounds. Correct repairs rise from 16 with the basic prompt to 58 after dialogue, but some faults remain unrepaired or are changed back to faulty.
5 RQ3. EFFECTIVENESS OF CHATGPT’S DIALOGUE
- The dialogue process first adds benchmark-provided fault locations, then supplies locations of faults still repaired incorrectly for another round.The study applies these prompts to cases not completely repaired after the enhanced-prompt stage.Source: §5 · pp. 6–7
- Some repairs revert to faulty in later rounds, which the authors attribute to ChatGPT repeatedly revising repairs without confidence.Fourteen faults remain unrepaired even after location hints.Source: §5 · p. 7
- ChatGPT can ignore supplied locations and address other code instead; in one example, it retains unsuitable loss and activation functions by misunderstanding a binary-classification setup.The example uses data generated with make_multilabel_classification while setting the number of classes to two.Source: §5 · p. 7
- Correctly repaired faults increase from 16 with PT-B to 43 with PT-E, then 55 with location hints and 58 after hints for remaining faults.Even after the dialogue rounds, 14 faults remain unrepaired.Source: Figure 2 · p. 7, §5 · p. 7
6 CASE STUDY
The case studies illustrate ChatGPT’s code-smell handling and API assistance, including both useful repairs and limitations in recognizing API evolution. They show examples of naming and comment edits, dead-code removal, and API recommendations.
6 CASE STUDY
- The case-study examples use real ChatGPT answers, streamlined to show only relevant code and text.Source: §6 · p. 7
6.1 Code Smell and Refactoring
- ChatGPT can identify code smells such as dead code and redundant imports, and address them through refactoring.The study also notes that some refactoring, such as variable renaming, can cause crashes when changes are inconsistent.Source: §6.1 · p. 7
6.1.1 Naming Convention.
- ChatGPT suggests more descriptive variable names, with its X_train suggestion indicating it may infer that X represents training data.Source: §6.1.1 · p. 8
6.1.2 Comments.
- ChatGPT addresses comment smells by removing irrelevant comments and adding comments that explain code functionality.It identifies a mismatched #fix1 comment as a leftover and adds comments to distinguish data generation, model construction, and training.Source: §6.1.2 · p. 8
6.1.3 Dead Code.
- ChatGPT identifies unused and duplicated imports as dead code and removes them, sometimes abbreviating numpy as np.Source: §6.1.3 · p. 8
6.2.1 API Evolution.
- ChatGPT can flag deprecated APIs and suggest replacements, but its API knowledge is limited to libraries represented in training data through September 2021.The study describes replacing fit_generator with fit and changing the Keras parameter nb_epoch to epochs; parameter changes can also escape detection or trigger false repair alarms.Source: §6.2.1 · pp. 8–9
6.2.2 API Misuse.
- ChatGPT can address API misuse through refactoring or suggestions, including API simplifications and recommendations based on inferred code intention.The examples include replacing loc with iloc for integer-indexed data slicing and suggesting an evaluation step after training.Source: §6.2.2 · p. 9
6.2 API Usage
- API-related assistance includes both detecting misuse and recommending more appropriate API forms or parameter settings.The examples include using named parameters and consolidating separate layers into an API parameter, as well as recommending verbose=1 for progress updates.Source: §6.2.2 · p. 9
7 DISCUSSION
ChatGPT can generate a prompt semantically close to a human-written one, while an individual failing-test hint may not complete DL program repair.
7.1 Automated Prompt Template Filling
- The automated prompt template asks ChatGPT to infer the problem type, dataset, and repair goal from a description and insert them into a repair request.The example fills these fields with classification, MNIST, and improving the network’s learning.Source: §7.1 · p. 9
- ChatGPT’s filled prompt had almost the same meaning as the human-written reference, indicating that prompt filling could be automated.The proposed few-shot template fills in the problem type, dataset, and repair goal from a natural-language description.Source: §7.1 · p. 9
7.2 Failing-Test-Guided Prompt
- Fault-location hints enabled ChatGPT to repair three of four faults in the XOR example, whereas the earlier prompts repaired at most one.A failing test still exposed an unrepaired fault after the location hints.Source: §7.2 · p. 10
- An individual failing-test hint did not fix the remaining XOR fault, and the authors conclude such hints may not help DL repair because decision logic is not explicit and ChatGPT cannot execute programs.Even after receiving the failing input and expected output [0], ChatGPT added a rounding step but still could not repair the fault; patches therefore need validation.Source: §7.2 · p. 10
8 RELATED WORK
The related work covers LLMs for software engineering and approaches tailored to DL program repair, positioning this study among both areas.
8.1 Large Language Model for Software Engineering (LLM4SE)
- Zeng et al. found pre-trained LLMs more promising than non-pre-trained models across evaluated code-understanding and code-generation tasks.Their study evaluated eight pre-trained LLMs and four non-pre-trained models across three code-understanding and four code-generation tasks.Source: §8.1 · p. 10
- This study complements Sobania et al. [40] by examining more complex programs and dependencies, comparing prompt templates, and summarizing ChatGPT’s response pros and cons.Sobania et al. evaluated ChatGPT on conventional Python programs with basic functionalities such as sorting and searching.Source: §8.1 · p. 10
8.2 Deep Learning Program Repair
- DL repair approaches include runtime monitoring, built-in debugging heuristics, fault localization, and diagnosis methods that can suggest repairs.AutoTrainer targets five training problems, while DeepLocalize locates symptoms in layers rather than the fault’s root cause.Source: §8.2 · p. 10
9 THREATS OF VALIDITY
The authors discuss interface variation, manual evaluation subjectivity, and possible benchmark-data leakage as threats to validity.
9 THREATS OF VALIDITY
- Repeating RQ1 with the ChatGPT website interface (Mar 14, 2023 Version) produced only a slight difference from the API experiment.The authors released the evaluation results for reference.Source: §9 · p. 10
- Because ChatGPT’s replies were carried out manually, the evaluation may involve subjectivity and misunderstandings.The cases include StackOverflow post IDs, and prompts and replies are traceable in the released repository.Source: §9 · p. 10
- Benchmark cases may have appeared in ChatGPT’s training data, but poor performance across prompt templates partially alleviates the authors’ memorization concern.The authors present this as a partial alleviation, not proof that leakage did not occur.Source: §9 · p. 10
10 CONCLUSION
The conclusion presents a study of ChatGPT for DL debugging, prompt effects, and dialogue, and identifies possible directions for supporting software engineering.
10 CONCLUSION
- The study examines ChatGPT’s DL debugging ability across fault detection, fault localization, and program repair.It also explores prompt effects and how dialogue might further improve debugging performance.Source: §10 · p. 10
- The authors propose a prompt template intended to improve ChatGPT’s DL debugging performance.The conclusion also examines how far dialogue can push performance.Source: §10 · p. 10
- The paper summarizes pros and cons of ChatGPT’s replies and points to possible ways it could facilitate the software-engineering community.Source: §10 · pp. 10–11