PaperlayerView exact revision on arXiv

Source-linked AI summary

MPI-rical: Data-Driven MPI Distributed Parallelism Assistance with Transformers

Nadav Schneider, Tal Kadosh, Niranjan Hasabnis, Timothy Mattson, Yuval Pinter, Gal Oren

arXiv:2305.09438v3cs.DCcs.CLcs.LG

TL;DR

Manually writing MPI domain-decomposition code is challenging, and translating serial code into distributed-memory code remains an unaddressed task in the cited work. MPI-RICAL fine-tunes a Transformer-based code model on MPICodeCorpus to suggest MPI functions and their locations; the authors report good performance on corpus tests and real-world numerical-computation programs.

  • Problem

    Manual MPI domain decomposition is challenging, while the cited work reports no developed tool for translating serial code into distributed-memory code.

    Source: §I · p. 1

  • Method

    MPI-RICAL fine-tunes SPT-Code on a task-specific dataset from MPICodeCorpus to suggest MPI functions and code locations.

    Source: §I · p. 2

  • Results

    MPI-RICAL performs well on MPICodeCorpus test programs and on 11 compiled MPI domain-decomposition programs representing real-world numerical computations.

    Source: §VII · p. 8

  • Takeaways & Limitations

    The authors conclude that the results indicate MPI-RICAL has good domain-decomposition and MPI-based parallelization understanding.

    Source: §VII · p. 8

Abstract

from arXiv · show

Message Passing Interface (MPI) plays a crucial role in distributed memory parallelization across multiple nodes. However, parallelizing MPI code manually, and specifically, performing domain decomposition, is a challenging, error-prone task. In this paper, we address this problem by developing MPI-RICAL, a novel data-driven, programming-assistance tool that assists programmers in writing domain decomposition based distributed memory parallelization code. Specifically, we train a supervised language model to suggest MPI functions and their proper locations in the code on the fly. We also introduce MPICodeCorpus, the first publicly available corpus of MPI-based parallel programs that is created by mining more than 15,000 open-source repositories on GitHub. Experimental results have been done on MPICodeCorpus and more importantly, on a compiled benchmark of MPI-based parallel programs for numerical computations that represent real-world scientific applications. MPI-RICAL achieves F1 scores between 0.87-0.91 on these programs, demonstrating its accuracy in suggesting correct MPI functions at appropriate code locations.. The source code used in this work, as well as other relevant sources, are available at: https://github.com/Scientific-Computing-Lab-NRCN/MPI-rical

I. INTRODUCTION

Distributed-memory simulations use MPI and domain decomposition to manage large computations, but manually placing communication calls in serial code is difficult. MPI-RICAL proposes an in-editor, data-driven assistant for suggesting MPI functions and their locations, supported by a newly created MPI program corpus and evaluations on corpus and numerical-computation programs.

I. INTRODUCTION

  • Distributed simulations divide data structures into subdomains that run the same computations in parallel, reducing the time needed for complex simulations.High-resolution simulations can require more memory and computation than a single node provides.Source: §I · p. 1
  • Manually parallelizing serial code for distributed memory is difficult because MPI send/receive calls can be misplaced and correct placement requires understanding dependencies and code structure.The challenge is especially acute in large source files or when the programmer is unfamiliar with the code.Source: §I · pp. 1–2
  • MPI-RICAL suggests MPI functions and insertion locations to assist programmers with domain-decomposition parallelization.The authors position it as an IDE-integrable assistant rather than a tool that directly generates MPI programs from natural-language prompts.Source: §I · p. 2
  • MPI-RICAL is trained and evaluated on 25,000 GitHub C programs and is also evaluated on numerical-computation programs with domain-decomposition code.The authors report that it performs well on both the test dataset and the numerical-computation benchmark.Source: §I · p. 2
  • The paper presents MPI-RICAL as, to the authors’ knowledge, the first data-driven programming-assistance approach for distributed-memory systems, specifically MPI.Its motivation includes the lack of tools translating serial code into distributed-memory code and the complexity of domain decomposition.Source: §I · pp. 1–2
  • MPICodeCorpus provides domain-decomposition programs and corresponding MPI-parallel programs, mined from 16,500 GitHub repositories.The authors describe it as the first publicly available corpus of MPI-based parallel programs.Source: §I · p. 2

II. PREVIOUS WORK

The paper situates its work against automated parallelization, noting that translating serial code to distributed-memory code had not been addressed by the cited approaches.

II. PREVIOUS WORK

  • Automated translation from serial code to distributed-memory code had not been developed, despite active work on parallelization tools.The passage contrasts this gap with translation approaches for shared-memory systems.Source: §II · p. 2
  • The cited prior work includes tools for translating serial code into shared-memory parallel code.Source: §II · p. 2
  • The passage frames serial-to-distributed-memory translation as distinct from the established shared-memory translation area.Source: §II · p. 2

A. Serial to Shared-Memory Parallelization

Prior parallelization research spans rule-based and data-driven approaches for shared memory, while distributed-memory work addresses OpenMP conversion and natural-language-based MPI generation rather than serial-to-MPI translation.

A. Serial to Shared-Memory Parallelization

  • Data-driven language models have been applied to shared-memory parallelization and can handle incomplete code and support code-editor assistance.Earlier shared-memory tools also include heuristic or rule-based compilers such as Cetus and Par4All.Source: A. Serial to Shared-Memory… · pp. 2–3

B. Serial to Distributed-Memory Parallelization

  • OpenMP-to-MPI conversion exists, but it is simpler than serial-to-MPI translation because OpenMP has already identified structured parallelizable blocks.MPI domain decomposition also requires distributing memory and placing send/receive calls with knowledge of code dependencies.Source: B. Serial to Distributed-Memory… · p. 3
  • Natural-language-to-MPI approaches include HPC-Coder, but its evaluation used well-known problems with existing MPI implementations, leaving performance on unseen problem descriptions unclear.The paper distinguishes this generation task from assisting programmers as they implement domain decomposition in serial code editors.Source: B. Serial to Distributed-Memory… · p. 3

III. RESEARCH OBJECTIVES

The research objectives assess whether MPI-RICAL can produce appropriate MPI function calls and place them correctly. The approach fine-tunes SPT-Code on a task-specific dataset derived from MPICodeCorpus.

III. RESEARCH OBJECTIVES

  • The study asks whether MPI-RICAL generates correct MPI functions and inserts them at correct code locations.For function correctness, the evaluation checks function names rather than arguments; locations are treated as source-code line numbers.Source: §III · p. 3
  • The function-correctness evaluation checks which MPI function names are called, without evaluating their arguments.The location question is separately assessed using source-code line numbers as the location unit.Source: §III · p. 3

IV. MPI-RICAL

  • MPI-RICAL is created by fine-tuning SPT-Code on a task-specific dataset from MPICodeCorpus because the base model does not support C/C++ or HPC programs.Source: §IV · p. 3

A. A brief background on SPT-Code

SPT-Code combines source code with a linearized AST representation, while MPI-RICAL applies it to predict MPI functions and their locations in code. The task is framed as code translation from MPI-free code to code containing MPI functions.

A. A brief background on SPT-Code

  • SPT-Code takes plain code, linearized AST, and natural language as input components, and its decoder is pretrained alongside its encoder.The model uses X-SBT to represent AST structure in a linearized form.Source: A. A brief background on SPT-Code · p. 3
  • X-SBT retains syntactic information at expression-level nodes and above in an XML-like sequence, reducing traversal length by more than half.It addresses ambiguity in lossy classical AST traversals and the length of SBT sequences, which can exceed the original code by more than three times.Source: A. A brief background on SPT-Code · p. 4

B. Fine-Tuning — Code Translation

  • MPI-RICAL treats translation as generating MPI functions and their locations from code without MPI functions, using source code and its linearized AST as encoder input.The source code and linearized AST are separated by a [SEP] symbol.Source: B. Fine-Tuning — Code Translation · p. 4
  • The MPI-RICAL workflow prunes MPI calls from corpus programs to create inputs, while the original programs provide labels for the translation task.The workflow also generates X-SBT representations for the inputs.Source: B. Fine-Tuning — Code Translation · p. 4

C. Measuring performance of MPI-RICAL on the research objectives

The evaluation framing treats MPI-function prediction and prediction of function locations as classification objectives. The supplied workflow describes removing MPI functions and their locations before training a translation model.

C. Measuring performance of MPI-RICAL on the research objectives

  • The workflow removes MPI functions and their original locations from code and X-SBT representations before using the resulting subset to train SPT-Code.The model predicts the desired MPI classification for new samples in those forms.Source: Figure 2 · p. 5
  • Table I is presented as containing general statistics related to MPICodeCorpus.Source: Table I · p. 6
  • RQ1 is framed as predicting one of 456 MPI functions, while RQ2 predicts whether an MPI function appears after a given code token.RQ1 is multi-class classification and RQ2 is binary classification.Source: C. Measuring performance of MPI-RICAL… · pp. 4, 6

V. MPICodeCorpus

MPICodeCorpus was built by mining GitHub repositories for C programs containing MPI-related terms and selecting complete, parseable programs. Hardware and parsing constraints narrowed the examples used for training.

A. Corpus

  • The corpus extraction yielded 59,446 C programs from approximately 16,500 repositories selected using “MPI” in repository metadata or README files.A program is defined as a source file containing the main function, headers, and implementation files.Source: A. Corpus · p. 6

1) Inclusion Criteria:

  • Programs were parsed with pycparser, and only complete programs that parsed successfully were retained.The stated aim was to include complete MPI programs with their relevant MPI function calls.Source: 1) Inclusion Criteria · p. 6

2) Exclusion Criteria:

  • Training examples were limited to files of 320 tokens, approximately 50 lines, and nearly half of the corpus examples were dropped.Programs that pycparser could not parse were also excluded.Source: 2) Exclusion Criteria · p. 6

3) Code Standardization:

The dataset standardizes source formatting and pairs MPI-free code inputs with original MPI code labels. Corpus statistics also characterize the extent of parallel code within programs.

3) Code Standardization:

B. Dataset Creation

  • MPI function calls are removed from input programs, while the corresponding original MPI-based programs serve as labels in a dataset of approximately 25,000 examples.Figure 4 depicts extracting programs, selecting them, removing MPI calls from successful snippets, and adding the resulting snippets to the dataset.Source: B. Dataset Creation · p. 6, Figure 4 · p. 7

C. Statistics

  • Most MPI programs have more than half of their lines inside the parallelized region, measured between MPI_Init and MPI_Finalize.Both functions appeared in 20,228 files in the raw data.Source: C. Statistics · pp. 6–7

VI. EXPERIMENTAL RESULTS

MPI-RICAL was trained and evaluated as a supervised classification task, with performance measured using standard classification metrics and a one-line location tolerance.

VI. EXPERIMENTAL RESULTS

  • MPI-RICAL was trained on 24,125 examples using an 80:10:10 train-validation-test split and five epochs.Training used a batch size of 32 and 320 tokens.Source: §VI · p. 7
  • Figure 5 plots training loss, validation loss, and accuracy across epochs.Source: Figure 5 · p. 7

A. Performance Metrics

  • Performance was measured using F1 score, precision, and recall.Source: A. Performance Metrics · p. 7
  • A generated MPI function matches ground truth only when its location is within one line; true negatives are out of scope.The study focuses on MPI function generation.Source: A. Performance Metrics · p. 7

B. Results on MPICodeCorpus

MPI-RICAL was evaluated on MPICodeCorpus and a benchmark of compiled numerical-computation programs. On the compiled benchmark, it achieved M-F1 of 0.91, M-Precision of 0.98, and M-Recall of 0.86.

B. Results on MPICodeCorpus

  • MPICodeCorpus results were evaluated separately for all MPI functions and the commonly used MPI Common Core functions, using M-F1 and MCC-F1.Source: B. Results on MPICodeCorpus · p. 7

C. Results on Fully Compiled Numerical Computations

VII. CONCLUSION AND FUTURE WORK

MPI-RICAL suggests MPI functions and their locations for domain-decomposition parallel code, using MPICodeCorpus for training. The authors report performance on test programs and compiled numerical-computation programs, while identifying structural code changes as outside the current evaluation.

VII. CONCLUSION AND FUTURE WORK

  • MPI-RICAL suggests MPI functions at appropriate locations for domain-decomposition parallel code, trained on the first MPI-specific code corpus, MPICodeCorpus.Source: §VII · p. 8
  • The authors report that MPI-RICAL performs well on 11 carefully selected, compiled MPI programs with domain decomposition.Source: §VII · p. 8
  • The current evaluation ignores structural code changes; future work also targets complete codes without length restrictions and accuracy of MPI function arguments and code structure.Source: §VII · p. 8

Related papers

Loading 2305.09438v3…