Improving MPI Error Detection and Repair with Large Language Models and Bug References

1. Introduction

• MPI errors can be difficult to isolate because nondeterministic behavior and changes in processing-node count can produce differing outcomes across executions.MPI’s non-blocking communication, wildcard operations, and collective operations requiring precise ordering add to this challenge.Source: §1 · p. 2
• Traditional static, dynamic, symbolic-execution, and concolic techniques cover only subsets of MPI defects and face false positives, scalability problems, or incomplete path coverage.These limitations make comprehensive detection in real-world MPI applications impractical.Source: §1 · p. 2
• The study proposes an approach combining CoT reasoning, few-shot learning, and RAG to improve LLM detection and repair of MPI defects.It compares improvements across ChatGPT, Llama2, QWen2.5-coder, and Code Llama rather than comparing the models themselves.Source: §1 · p. 3
• The approach improves MPI defect-detection accuracy over direct ChatGPT use by 75%.The study also evaluates ChatGPT’s ability to repair defective MPI code and tests applicability to other LLMs.Source: §1 · p. 3
• The evaluation focuses on known defective MPI programs from a public dataset, including cases where traditional analysis techniques had high false-negative rates.The authors describe this as the first effort to target LLM detection and repair for MPI defects that traditional methods struggle to identify.Source: §1 · p. 3

2. Related Work

• MPI’s non-deterministic communication and synchronization make defect detection and debugging challenging.MPI supports scalable parallelism in scientific simulations and distributed machine learning.Source: §2 · p. 3

2.1. Classic Error Detection and Repair in MPI Programs

• Static, dynamic, symbolic-execution, and concolic methods address selected MPI defects, but face false positives, runtime or scalability costs, incomplete coverage, and limited applicability to large programs.MPI-CHECK and instrumentation approaches address particular issues; dynamic methods analyze exercised paths, while symbolic approaches encounter state-space explosion.Source: §2.1 · p. 4

2.2. Modern Software Defect Detection and Repair with LLMs

• LLM work spans fault localization, analyzer-assisted detection, test generation, and repair, with approaches often limited by context or auxiliary artifacts, narrow defect scope, prompt sensitivity, or domain-knowledge needs.MPI-specific faults such as deadlocks, race conditions, and collective mismatches remain underexplored in this literature [26] [30] [31].Source: §2.2 · p. 5

3.1. LLM Configuration

• ChatGPT queries were repeated five times in independent conversations to mitigate nondeterministic model behavior, and the protocol was replicated with Llama2, Qwen2.5-Coder, and Code Llama.The open-source models were selected for availability and parameter sizes comparable to GPT-3.5 Turbo, approximately 13 billion parameters.Source: §3.1 · p. 6

3.2. Dataset Pre-processing

• The MPI Bugs Initiative samples’ source-code comments were removed because they may contain defect-descriptive annotations uncommon in real-world MPI code.Comment removal was intended to prevent those annotations from influencing the analysis.Source: §3.2 · p. 6

4. Bug Detection of MPI Programs

• The section introduces defect-detection terms relative to whether a defect exists and whether the analysis reports one, then frames targeted prompting as a response to MPI detection challenges.The evaluation focuses on complex MPI-specific defects that may lack direct analogs in serial programming.Source: §4 · p. 6, §4.1 · p. 7

4.2. Baseline with Zero-Shot

• ChatGPT’s zero-shot baseline had a 46.88% false-negative rate, missing 113 of 241 samples on average.Five trials assessed accuracy, precision, recall, and F1 score.Source: §4.2 · pp. 7–8

4.3. Enhancing defect detection via explicit bug referencing

• The study combines Few-Shot Learning, CoT, and RAG with a domain-informed repository of MPI bug examples spanning resource leaks, communication errors, invalid parameters, and synchronization defects.The repository and expert guidance support evaluation of prompting combinations.Source: §4.3 · p. 8

4.3.1. Incorporating Bug References via Few-Shot Prompts

• Few-Shot bug examples raised accuracy by 11.95%, from 44.39% to 56.34%, over zero-shot prompting.The examples pair defective MPI code with expert explanations.Source: §4.3.1 · pp. 9–10, Table 2 · p. 10

4.3.2. Incorporating bug references with Chain-of-Thought

• Few-Shot+CoT improved accuracy by 33 percentage points over zero-shot, with an average false-negative rate of 1.5% across five trials.The prompts guide step-by-step checks for common MPI errors.Source: §4.3.2 · pp. 11–12

4.3.3. The Alternative - RAG with Good and Defective MPI Examples

• RAG improved detection accuracy over zero-shot but did not match Few-Shot+CoT; its retrieval databases varied the ratio of correct to defective MPI examples.The authors attribute RAG’s limited effectiveness to unannotated defective examples, code-similarity limitations, and sensitivity to chunking and retrieval noise.Source: §4.3.3 · pp. 14–15

4.4. Statistical Validation

• Recall rose from 0.35 to 0.98 and accuracy from 0.44 to 0.78, while specificity fell from 0.69 to 0.26 against zero-shot; McNemar’s test found the shift significant (p < 0.001).Precision remained stable (0.75 to 0.77), and F1 rose from 0.47 to 0.86.Source: §4.4 · p. 16

4.5. Evaluation of non-LLM tools

• The bug-referencing approach achieved a 2.3% miss rate, approximately 15× below the non-LLM tools’ 34.75% average, and a 10% lower false-positive rate than the best non-LLM tool.The comparison covers CIVL, ISP, MPI-SV, and PARCOACH.Source: §4.5 · p. 17, Table 7 · p. 17

5.2. Repair Evaluation Metrics

• Repairs were tested for compilation, resource-leak removal, and deadlock removal, with automated checks for leaks and executions exceeding 900 seconds.The benchmark comprised 164 programs consistently flagged as defective across five Few-Shot+CoT trials.Source: §5 · p. 17, §5.2 · p. 18

5. Automated MPI Program Repair

• The authors report no established repair baseline because this was their first measurement of ChatGPT’s ability to detect and repair MPI defects.The repair prompt combined Few-Shot bug references, Chain-of-Thought reasoning, and the MPI code.Source: §5.3 · p. 18, §5 · p. 18

5.3. Repair Analysis

• 92.68% of generated repairs compiled successfully; the 12 failures involved variable redeclarations, missing statement terminators, or incorrect MPI argument types.The latter errors were absent from the original programs and were attributed to hallucinations or unintended modifications.Source: §5.3 · pp. 18–19
• After excluding MPI-library leaks, ChatGPT repaired 11 of 13 known application-level leaks, an 84.76% success rate.One failed repair contained a missing semicolon.Source: §5.3 · p. 19
• ChatGPT successfully repaired 76.83% of deadlock defects in the dataset.The study identifies deadlocks as a common MPI error class.Source: §5.3 · p. 19
• 76.21% of repairs (125 of 164) met all three success criteria; Figure 4 summarizes success and failure rates by defect type.A repair counted as successful only if it compiled, removed resource leaks, and avoided deadlocks.Source: §5.3 · p. 19, Figure 4 · p. 20
• Most repair failures arose from parameter-matching defects that cause execution deadlocks and require aligning communication parameter types and tag values.The authors note that resolving these defects requires understanding MPI call intent beyond basic syntax checks.Source: §5.3 · p. 19

6.1. Experimental Setup

• The authors tested Llama2, Code Llama, and Qwen2.5-coder using the same trials as ChatGPT, with the models in their default configurations.The three models were selected to assess applicability beyond ChatGPT, including open-source and code-specific models.Source: §6 · p. 20, §6.1 · p. 21

6.2. Classification Performance Metrics

• Bug referencing produced the best accuracy, precision, recall, and F1 MPI defect-detection scores across all three models.It also yielded the largest true-positive increases and false-negative decreases; Figures 5 and 6 compare the prompting strategies.Source: §6.2 · p. 21, Figure 5 · p. 21, Figure 6 · p. 22

6.3. Statistical Validity

• McNemar’s tests showed statistically significant improvements over zero-shot baselines for all three models, which the authors interpret as improved discriminative capability rather than a threshold shift.Table 8 reports the test results.Source: §6.3 · p. 22, Table 8 · p. 22

6.4. Cross-LLM Performance Stability

• Bug referencing had the lowest variance in true positives, true negatives, false positives, and false negatives across the three models and prompting techniques.Table 9 summarizes the variance comparison.Source: §6.4 · p. 23, Table 9 · p. 23

6.5. Defect Repair Performance

• Qwen2.5-coder achieved the highest successful-compilation, leak-removal, and deadlock-removal rates among the three models in the repair trials.Figure 7 shows repair success and failure rates by defect type.Source: §6.5 · p. 23, Figure 7 · p. 24

6.6. Summary

• Adding retrieval augmented generation degraded performance across all three models, repeating the trend observed in the ChatGPT trials.The authors cite missing defect annotations, brittle token-level similarity, chunking, and retrieval noise as contributing constraints, especially for MPI calls differing in communicators, ordering, or tags.Source: §6.6 · pp. 24–25
• ChromaDB retrieval distances ranged from 0.119176 to 0.129195 across the three models, with an approximately 0.01 spread described as stable and low-distance.Figure 8 presents retrieval trends for Llama2, Code Llama, and Qwen2.5-coder.Source: §6.6 · p. 25, Figure 8 · p. 25

7.1. Internal

• LLM non-determinism can produce varying outputs for the same input; the authors ran repeated trials with the same prompts and data and reported trial averages.They also could not rule out ChatGPT training-data overlap with test programs because the training-set composition is unavailable.Source: §7.1 · pp. 25–26

7.2. External

• Findings from the publicly available dataset may not generalize to other datasets, programming paradigms, or LLMs, and model availability or updates pose repeatability risks.The authors note that ChatGPT GPT-3.5 Turbo may be discontinued and that open-source models undergo retraining and replacement.Source: §7.2 · p. 26

• 84.62% of resource leaks and 83.44% of deadlocks were corrected by ChatGPT, versus 67.39% of mismatched parameter tag defects.Source: §8 · p. 26
• Bug references improve ChatGPT’s detection accuracy over its out-of-the-box configuration; domain-expert–guided Chain-of-Thought reasoning adds gains and reduces variance across true and false positives and negatives.Source: §8 · p. 26
• The approach improves detection of hard-to-find MPI errors in datasets where state-of-the-art tools have high false-negative rates.Source: §8 · p. 27

• LLMs frequently failed to correctly repair deadlock defects, motivating future prompts that better convey MPI program intent during repair.Source: §9 · p. 27
• RAG database composition had minimal impact on detection accuracy, while text-based embeddings struggled to capture MPI concurrency, message-passing semantics, and communication hazards.Source: §9 · p. 27
• The authors identify code-aware representations and retrieval methods tailored to MPI program structure as needed directions, alongside domain-expert guidance.Source: §9 · p. 27

Appendix A. Experimental Trial Results for ChatGPT

• Appendix A is titled as experimental trial results using ChatGPT.Source: Appendix A · p. 34

Appendix B. Experimental Trial Results for Llama2

• Appendix B identifies experimental trial results using Llama2.Source: Appendix B · p. 35

Appendix C. Experimental Trial Results for Code Llama

• Appendix C identifies experimental trial results using Code Llama.Source: Appendix C · p. 36

Appendix D. Experimental Trial Results for QWen2.5-coder

• Appendix D identifies experimental trial results using QWen2.5-coder.Source: Appendix D · p. 38

Appendix E. Sample code

• Appendix E illustrates a resource leak from missing MPI resource-release calls and a mismatched tag that leaves Rank 1 waiting indefinitely.Source: Appendix E · pp. 40–41