PaperlayerView exact revision on arXiv

Source-linked AI summary

Harnessing MPI mutations for AI error detection

Asia Auville, Tim Jammer, Eric Petit, Pablo de Oliveira Castro, Emmanuelle Saillard, Mihail Popov

arXiv:2610.20616v1cs.SEcs.DC

TL;DR

MPI usage errors are difficult to detect, and existing AI methods lack reliable real-world MPI evaluation data. The paper builds a mutation-based MPI dataset and trains and evaluates AI detectors; its embedding-based models reach 91.5% average accuracy across MPI error types on mutated validation splits.

  • Problem

    MPI error detection remains difficult to benchmark, and existing datasets lack combinations of representative code, reliable labels, and useful context.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    The paper applies realistic MPI-specific mutations to GitHub-mined MPI programs and uses the labeled dataset to train and evaluate embedding-based classifiers and LLM approaches.

    Source: §1 · p. 2, §4.2 · p. 5

  • Results

    91.5% average accuracy across all MPI error types was achieved on mutated validation splits using the MBB classification.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The mutation-based dataset offers practitioners examples for evaluating verification methods and training AI detectors, shifting effort from hand-coded expert rules toward labeling or generating data.

    Source: §6 · p. 12, §7 · p. 13

    MPI code datasets can lack cross-process context, and commit-message-based labeling can misidentify an error’s root cause.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

MPI errors are challenging to identify despite the significant number of expert verification tools. Dynamic tools (i.e., requiring profiling) are computationally expensive and accurate in error detection, whereas static analysis (i.e., operating at source code or compilation) is computationally cheap but less accurate. Interestingly, the recent success of AI and LLMs offers an alternative to increase static analysis accuracy while preserving its low overhead. Yet current methods remain difficult to benchmark, too general, and poorly adapted to the specific challenges of high-performance computing. In this paper, we investigate how AI-powered tools can efficiently and accurately detect errors in real-world MPI applications. We propose a novel MPI Mutated Dataset (MMD), constructed from MPI programs extracted from thousands of open-source GitHub projects. After sorting and filtering these files, we inject errors that realistically emulate developers' mistakes using synthetic code mutations. We leverage the dataset to train different AI models and assess their generalization capabilities against standard verification tools. We train compact multilayer perceptrons (MLP) with AST-T5 embeddings (i.e., transformer only encoder network) to detect diverse MPI errors (from the MPI-BugBench (MBB) classification) including CallOrdering or InvalidParameter, achieving detection accuracies up to 92.0%. In some cases, our method outperforms the expert prompted Qwen-3 Coder LLM and provides competitive results compared to established dynamic verification tools. We further evaluate Claude Sonnet 4.5 across different prompting strategies, showing that engineered prompts improve InvalidParam detection accuracy from 84.8% to 93.6%, surpassing the dynamic tool ITAC (87.0%). The full agentic pipeline achieves 73.9% overall multi-class accuracy and 87.6% bug detection recall on MBB. This paves the way for large-scale verification efforts that can use our dataset as a foundation to thoroughly investigate each MPI error in real world applications.

1 Introduction

MPI error detection is difficult, and existing AI methods lack reliable real-world MPI evaluation datasets. The paper introduces realistic mutations and a dataset for training and evaluating AI detectors, reporting up to 91.5% average accuracy across MPI error types on mutated validation splits.

1 Introduction

  • Existing tools cannot detect every error, while AI methods lack reliable real-world MPI detection and established evaluation datasets.Prior models used non-specialized codebases or were evaluated on small systematic datasets.Source: §1 · p. 1
  • The paper proposes realistic synthetic MPI bug injection and a large-scale dataset for evaluating verification tools and AI methods.It trains supervised classifiers from AST-T5 or IR2VEC code embeddings paired with error labels.Source: §1 · p. 2
  • 91.5% average accuracy across all MPI error types was achieved on mutated validation splits using the MBB classification.The paper also reports comparisons across code origins and against dynamic verification tools.Source: §1 · p. 2
  • The study evaluates embedding-based models, standard verification tools, and LLMs using different prompt strategies, including an agentic pipeline.The paper identifies the dataset, mutation method, AST-T5 MLP, accuracy analysis, and prompt strategies as contributions.Source: §1 · p. 2

2 Related Works

Prior MPI verification and AI approaches face limits in scope, generalization, or evaluation data. The paper contributes MPI-specific mutations and a dataset, and studies AST-T5-based detection and prompt-engineered LLMs.

2 Related Works

  • Static verification can be scalable but limited by compile-time information, while dynamic tools cover more errors at higher execution and trace-analysis cost.PARCOACH may report false positives; MUST and ITAC require full MPI execution and depend on execution order and inputs.Source: §2 · p. 2
  • Prior MPI AI studies use same-dataset validation, synthetic-only codes, or short synthetic examples, leaving real-world generalization uncertain.The reviewed work reports 97% accuracy on hybrid MPI/OpenMP samples, while Few-Shot references and CoT raise accuracy from 44% to 77% on short synthetic codes [29].Source: §2 · p. 2
  • The paper applies AST-T5 to MPI verification and evaluates general LLMs with refined prompts, addressing code representation and task scope.ASTs and IR-based graphs are highlighted as structured representations relevant to code-detection accuracy.Source: §2 · pp. 2–3
  • MBB supplies labeled MPI correctness examples, whereas large MPI code collections such as MPICodeCorpus lack correctness labels for training.MBB contains 2,000 synthetic codes; MPICodeCorpus contains around 50,000 real-world programs scraped from GitHub [34].Source: §2 · p. 3
  • The paper presents MPI-specific source-code mutations and an augmented dataset as a basis for training AI models and benchmarking verification tools.Existing mutation methods lack MPI-specific mutations, while prior fault injection targeted runtime faults rather than erroneous MPI usage [9].Source: §2 · p. 3

3 Dataset Creation

Existing MPI datasets trade off labeling reliability, scale, and code representativity. The paper addresses these constraints with deterministic MPI-specific mutations applied to real-world code, yielding a labeled dataset for training and verification-tool evaluation.

3.1 Limitations of Existing Datasets

  • Existing MPI datasets compromise among label reliability, scale, and representativity because code context and reliable MPI error labels are difficult to obtain.Curated datasets can lack cross-process context and MPI-specific code; dynamic labeling has high overhead, while MBB is synthetic and structurally homogeneous.Source: §3.1 · p. 3

3.2 MPI Mutation Tool

  • The mutation tool implements 33 mutations across five of eight MBB error categories, using AST-based transformations to introduce MPI errors.Mutations deterministically introduce certified errors and preserve other code pieces when applicable; some categories require nondeterministic mutation.Source: §3.2 · pp. 3–4
  • Mutations cover errors including LocalConcurrency, CallOrdering, MissingCall, ParameterMismatch, and InvalidParam, with additional correct mutations to reduce artifact-based detection.The tool supports multiple mutation locations and can create mismatches by changing MPI arguments or inserting calls.Source: §3.2 · pp. 3–4

3.3 The MPI Mutated Dataset

  • MMD contains around 36,000 MPI codes: approximately 2,000 GitHub-mined base programs and about 34,000 mutated variants.Base programs are assumed correct; each mutation is labeled as erroneous or semantically correct, and correct mutations help avoid learning mutation artifacts.Source: §3.3 · p. 4
  • InvalidParam mutations match more AST nodes, producing more mutated samples and class imbalance in MMD.Figure 1 presents the label distribution; MMD labels GitHub-scraped code through mutations, unlike unlabeled MPICodeCorpus and explicitly labeled MBB.Source: §3.3 · p. 4, Figure 1 · p. 5, Table 1 · p. 5
  • Benchmarks are stripped of textual correctness or error cues, and MMD programs are removed from MPICodeCorpus to prevent overlap.MMD and MBB share an error classification system, enabling direct benchmark comparison.Source: §3.3 · p. 5

4 Bug Detection Models

The section establishes an embedding-to-classifier workflow for testing MPI error detection across datasets, alongside prompted LLM comparisons. It also describes an agentic prompt pipeline motivated by false positives from independent specific prompts.

4.1 Objective and workflow

  • The workflow extracts code embeddings for MLP training, then tests generalization through dataset splits or unseen datasets, while prompted LLMs provide a comparison.Training uses MMD or MBB; unseen-dataset tests include MBB or MCC.Source: §4.1 · p. 5, Figure 2 · p. 6

4.2 Embedding Methods

  • AST-T5 encodes raw source into flow- and structure-aware vectors without compilation, while IR2VEC uses LLVM IR and requires compilation.For oversized files, AST-T5 uses overlapping 1,024-token windows with 300-token overlap and averages their embeddings.Source: §4.2 · p. 5

4.3 Model Structure

  • The MLPs are lightweight six-layer feed-forward classifiers that take embeddings as input and predict error labels, with architecture choices tuned empirically.The described network uses decreasing hidden dimensions and ReLU, with dropout at 0.1 between the first four layers.Source: §4.3 · pp. 5–6
  • Training spans synthetic MBB baselines and MMD classifiers for specific error types or unified error detection, with error-free mutants included to reduce mutant-status bias.The MMD error categories include InvalidParam, ParameterMismatch, CallOrdering, MissingCall, and LocalConcurrency.Source: §4.3 · p. 6

4.4.1 Baseline Prompt Evaluation.

  • Initial prompts used category-specific or generic safe/unsafe questions; engineered Claude prompts were refined from MBB descriptions and confusion cases, with skepticism instructions to reduce false positives.The initial prompts were tested with Qwen-3 Coder and Claude Sonnet 4.5; engineered prompts were evaluated with Claude.Source: §4.4.1 · p. 6

4.4.2 Agentic Pipeline.

  • Independent prompts detected bugs or target errors, but Specific prompts also produced many false positives by assigning other error types to their target category.These observations motivated the multi-agent scheduling pipeline.Source: §4.4.2 · p. 7
  • The agentic pipeline screens codes with a Manager, routes suspected categories to skeptical Specialists, and invokes a Judge only for specified ambiguities.The Judge is skipped when all Specialists reject; a single confident confirmation is accepted directly.Source: §4.4.2 · p. 7

4.5 Model validation

  • Validation includes standard train–validation splits and cross-dataset tests; MLP stability uses five meta-repetitions, while costly LLM evaluations cover selected scenarios and report macro and threshold-independent metrics.ROC-AUC and PR-AUC supplement macro scores for imbalanced binary-classifier data.Source: §4.5 · p. 7

5 Experimental Results

The experiments assess how MPI error detectors generalize across datasets and compare AI methods with verification tools. Results identify both promising cross-dataset performance and constraints tied to synthetic errors, code length, and evaluation scope.

5.1 Limits of Current AI4MPI Verification

  • AST-T5 generalized more promisingly than IR2VEC, while models trained only on synthetic MBB codes overfit and failed to outperform random predictions on MMD and MCC.The MBB-only model reached 100% accuracy on MBB after a few epochs, indicating overfitting.Source: §5.1 · p. 8

5.2 Generalizing MPI Verification

  • InvalidParam MLPs achieved around 80% accuracy across MMD and MBB, but correct-code prediction showed a substantial F1 drop on MBB despite 86% accuracy on MMD.The model tended to predict the generic error label even for correct MBB codes.Source: §5.2 · p. 9

5.3 In Depth Accuracy Characterization

  • Training-data amount did not cap detection, whereas longer code and mutation-aware splits hurt performance; representation shift particularly challenged ParameterMismatch generalization to MBB.Most correctly predicted instances fell below the embedder’s 1,024-token context limit; mutation-aware splits underperformed random splits.Source: §5.3 · pp. 9–10

5.4 Manual validation of MLP predictions

  • Among 60 manually reviewed predictions, 29 of 50 codes flagged as erroneous contained an MPI error, and 19 had the exact error predicted.Twelve reviewed codes were duplicates, and nine contained no MPI function.Source: §5.4 · p. 11, Figure 8 · p. 11

5.5 LLMs

  • Engineered prompts raised Claude’s InvalidParam accuracy from 84.8% to 93.6% and reduced its false-positive rate from 18.1% to 4.3%.Recall decreased from 0.936 to 0.870, while F1 increased from 0.752 to 0.870.Source: §5.5 · p. 11, Table 5 · p. 11
  • 73.9% overall multi-class accuracy on MBB and 65.3% on MMD were achieved by the Claude Sonnet 4.5 agentic pipeline.On MBB, per-class recall ranged from 98.5% for LocalParamMismatch to 39.7% for LocalConcurrency.Source: §5.5 · p. 11, Table 6 · p. 11

5.6 Comparison against verification tools

  • For InvalidParam detection, engineered Claude prompts achieved 93.6% accuracy with balanced precision and recall (0.870), surpassing all other approaches, including ITAC at 87.0%.Source: §5.6 · p. 12

5.7 Energy Measurements

  • The largest MLP training configuration used about 24.4 Wh, and embedding inference used about 7.3 × 10^-3 Wh per 1,000 tokens, around 100 times less than a GPT-4o query.The MLP pipeline took around 0.1–1 seconds per sample, versus around 30–50 seconds for agentic LLM inference.Source: §5.7 · p. 12

6 Discussions

The paper shifts MPI error detection from manually encoded expert knowledge toward learning from examples, moving the costly requirement to labeling or generating training data. It also identifies non-determinism as a limitation and outlines possible uses of error predictions and datasets.

6 Discussions

  • AI learns error patterns from examples rather than requiring traditional tools’ manually implemented expert knowledge.This shifts the costly requirement to labeling or generating training data.Source: §6 · p. 12
  • The model returns a certainty ratio, while LLM outputs can depend on prompts and may not match the described error topology.The paper identifies non-determinism as a limitation of its approach.Source: §6 · p. 13
  • Predicted errors could help develop a more objective error definition by using latent text representations to quantify similarity between errors.The authors suggest using the information needed to detect errors as a guideline for classification.Source: §6 · p. 13
  • Future work could use the datasets to fine-tune LLMs to predict errors and fix code, or explore a hybrid approach using static and dynamic information.The proposed hybrid would provide compilation and verification information to models such as MLPs or LLMs.Source: §6 · p. 13

7 Conclusion

The paper investigates AI-based MPI usage-error detection using a dedicated mutation tool, a representative dataset, and alternative code embeddings. It presents the mutation tool, prompts, and dataset as resources for verification practitioners and AI research.

7 Conclusion

  • The work investigates AI-based detection of MPI usage errors using an alternative embedding and a diverse, representative dataset.A dedicated MPI mutation tool creates and augments the dataset to train more generalizable models.Source: §7 · p. 13
  • The mutation tool’s capabilities offer opportunities for MPI verification practitioners and broader AI research.The authors state that the tool, prompts, and dataset are available online.Source: §7 · p. 13
  • The paper frames its approach as an alternative for training more generalizable models.The conclusion connects this approach to combining alternative embeddings with a diverse and representative dataset.Source: §7 · p. 13
Loading 2610.20616v1…