PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring the Role of LLMs in HPC Programming: A Survey

Strahinja Ljaljevic, Josep Jorba, Sergio Iserte

arXiv:2608.26110v1cs.DCcs.AI

TL;DR

HPC programming is complex and specialized, while the evidence on how LLMs perform across HPC workflows remains uneven. This survey synthesizes research across five categories and finds that LLMs offer useful assistance but remain fragile for production HPC, making them collaborators rather than replacements for experts.

  • Problem

    Research lacks a comprehensive understanding of where LLMs succeed or fall short when supporting complex, specialized HPC programming workflows.

  • Method

    The survey systematically screens recent literature and organizes findings across code generation, parallelization and optimization, frameworks and architectures, evaluation, and broader challenges.

  • Results

    LLMs show useful potential for HPC assistance but remain brittle under uncompromising correctness, scalability, and performance requirements, and are unlikely to replace HPC experts soon.

  • Takeaways & Limitations

    Trustworthy LLM deployment in HPC will require richer datasets, integrated frameworks, rigorous evaluation, human validation, and governance structures.

  • Takeaways & Limitations

    Open, diverse HPC datasets and benchmarking standards that capture real-world complexity remain rare or underdeveloped.

Abstract

from arXiv · show

Large Language Models (LLMs) are emerging as promising assistants in High-Performance Computing (HPC), where programming remains complex and expertise-intensive. This survey systematically reviews their application across five categories: code generation, parallelization and optimization, frameworks and architectures, evaluation and benchmarking, and broader challenges. The analysis highlights both opportunities and limitations: while general-purpose LLMs perform reasonably well on serial and OpenMP-like tasks, they fall short in distributed paradigms such as MPI, where correctness and scalability are critical. Domain-specialized models (e.g., HPC-Coder, HPC-GPT, chatHPC) achieve higher accuracy through fine-tuning, curated datasets, and retrieval-augmented generation (RAG), yet their scope remains narrow and their evaluations largely limited to benchmarks or micro-kernels. The broader picture is one of dual potential and fragility: LLMs can lower barriers to entry, accelerate prototyping, and support code modernization, but they remain brittle under production-level requirements where correctness, performance portability, and scaling cannot be compromised. We conclude that LLMs are unlikely to replace HPC experts in the near term but are positioned to become powerful collaborators in the software development pipeline. Their effective deployment will require richer datasets, integration with performance analysis and schedulers, rigorous evaluation frameworks, and governance structures that ensure transparency and trust. The convergence of AI and HPC should therefore be understood as a long-term, co-evolutionary process, where each advance uncovers new challenges and opportunities for reshaping scientific software development.

1. Introduction

This survey examines how LLMs can support complex HPC workflows, including code generation, optimization, and system-level guidance. It synthesizes research through a structured review while identifying persistent gaps and directions for responsible AI-HPC integration.

  • Motivation: LLMs may lower HPC expertise barriers, automate repetitive coding, and support exploratory optimization for developers.They can assist with advanced programming paradigms and help developers focus on algorithmic design and performance-critical decisions.
  • HPC applications: LLM assistance can target diverse architectures by generating hybrid kernels, adapting cluster job scripts, and proposing transformations for memory locality and scalability.Examples include OpenMP, MPI, and CUDA environments.
  • Scope and organization: The survey covers HPC challenges across shared memory, distributed memory, and heterogeneous frameworks, organizing research into five thematic categories.The categories are code generation; parallelization and optimization; frameworks and architectures; evaluation and benchmarking; and broader challenges.
  • Methodology: The survey uses systematic searches across IEEE Xplore, ACM Digital Library, arXiv, and SpringerLink, emphasizing relevant, technically deep, recent research.The screening focuses on publications from 2023 onward and excludes studies lacking HPC context.
  • Challenges and future directions: Identified gaps in training data, model design, and usability motivate future work on GNN integration and interactive HPC-focused coding environments.The survey encourages deeper collaboration between AI and HPC communities to support responsible integration into scientific computing.

2. Background

LLMs extend language modeling to programming, enabling code generation and assistance across the HPC software lifecycle. In HPC workflows, they can support parallel programming, optimization, debugging, documentation, education, and modernization through integrated development tools and services.

  • LLM foundations: Transformer-based LLMs model language sequences and increasingly generate code, supporting tasks such as question answering, summarization, translation, and compute code generation.Their capabilities derive from modeling syntax, semantics, and context across token sequences.
  • Code LLMs: Code LLMs trained on source code, documentation, and natural-language descriptions can generate, interpret, and comment on programs across languages including C, C++, Python, and Fortran.This dual training connects natural-language descriptions with programming-language artifacts.
  • LLMs in HPC: In HPC, LLM applications span parallel code generation, performance optimization, debugging, refactoring, documentation, education, and porting legacy applications to heterogeneous hardware.Targeted tasks include MPI, OpenMP, CUDA, memory usage, vectorization, communication patterns, race conditions, deadlocks, and migration to GPUs, FPGAs, or QPUs.
  • HPC integration: HPC integration pathways include interactive notebooks, IDE plugins, assistants for MPI/OpenMP/CUDA programming, and chatbots that generate job scripts or configure software environments.A representative prompt asks for a hybrid MPI+OpenMP 3D stencil loop with minimal NUMA false sharing.

3. Methodology

This survey uses a structured, transparent review methodology to synthesize research on LLMs in HPC and identify questions guiding future investigation. It systematically searches and categorizes recent peer-reviewed literature while evaluating models, programming targets, evaluation strategies, and workflow integration.

  • The survey systematically synthesizes research at the intersection of LLMs and HPC to map the state of the art and derive future research questions.
  • A defined search strategy combines multiple academic databases, curated keywords, and explicit inclusion and exclusion criteria to support transparency, reproducibility, and extension.
  • The review focuses on peer-reviewed studies published from 2023 onward, when dedicated research on LLMs for HPC began to emerge.
  • Each selected paper is analyzed by LLM type, programming model, evaluation strategy, and depth of integration into HPC workflows.Programming models include OpenMP, MPI, GPU, task-based, and hybrid approaches; evaluation includes correctness, compilation, performance, scaling, testing, and expert validation.
  • The review’s research questions examine LLM uses, techniques, effectiveness, evaluation practices, comparisons with human-written code, challenges, and research gaps in HPC.

4. State of the Art: LLMs for HPC Programming

The surveyed literature shows growing interest in LLMs for HPC programming, with general-purpose models performing unevenly across programming paradigms and specialized workflows improving correctness, translation, and portability. However, limitations in data, task scope, performance, and human validation keep LLMs primarily in a collaborative role.

  • Research landscape: 32 of 72 screened scientific articles met the inclusion criteria, reflecting an emerging and increasingly active LLM–HPC research area.The studies span five thematic categories, and publication density has increased over time.
  • Code generation and parallelization: LLaMA-2 produced at least one correct implementation in 40% of C++, 66% of Fortran, 22% of Julia, and 33% of Python test cases.The evaluation covered AXPY, GEMV, and GEMM numerical HPC kernels across four programming languages.
  • Code generation and parallelization: LLMs perform better on serial-like and OpenMP tasks than on distributed parallel programming, with MPI posing particular challenges.The literature evaluates functional correctness and runtime-oriented measures including pass@1, pass@k, speedup, and efficiency.
  • Specialized models: Domain specialization substantially improves targeted HPC generation: HPC-Coder achieved a 97% success rate for functionally correct OpenMP pragmas.HPC-Coder also reached 92% accuracy on a coding competition dataset and 88% on another reported performance-prediction measure.
  • Specialized models: HPC-Coder-V2 outperformed open-source models by roughly 10-20 percentage points and achieved near GPT-4 performance across tested programming languages.Its gains came from task-specific fine-tuning and the synthetic HPC-INSTRUCT dataset, evaluated on the ParEval benchmark.

5. Conclusions

LLMs show dual potential and fragility in HPC: they can accelerate guided parallelization, modernization, and prototyping, but remain unreliable for production requirements involving correctness, scalability, and performance. Their effective deployment will require richer datasets, HPC-specific evaluation, human validation, integrated tools, and governance.

  • Conclusions: LLMs lower barriers to HPC, assist non-experts, and accelerate prototyping, yet remain brittle where correctness, scalability, and performance cannot be compromised.The literature characterizes this as a combination of dual potential and fragility.
  • Research agenda: Future progress requires high-fidelity HPC datasets, hybrid AI–HPC architectures, benchmarks covering runtime and scalability, interactive feedback, and human-in-the-loop validation.Recommended evaluation dimensions include memory footprint, energy consumption, debugging success rates, and correctness under parallel execution.
  • Conclusions: LLMs are unlikely to replace HPC experts soon but are poised to become powerful assistants within a software-development ecosystem requiring transparency, accountability, sustainability, and governance.The proposed ecosystem includes benchmarks, datasets, frameworks, traceability, deployment policies, and community consensus.
  • Contributions: LLMs can support guided parallelization, code translation, legacy modernization, profiling, portability, I/O, and performance engineering, but production-ready benefits remain concentrated in guided generation and translation.Applications include inserting OpenMP pragmas, translating legacy Fortran to C++, and proposing loop transformations.
  • Evaluation: Evidence at full-application scale remains scarce, with most studies focused on kernels or proxy applications and only limited case studies exceeding that scope.CodeScribe on a large Fortran LHC codebase is identified as an example beyond kernels or proxy applications.
  • Limitations: LLMs approach or match human correctness on narrow tasks, but ParEval reports efficiency <13% across models for parallel scaling, with MPI remaining a weak spot.The cited narrow tasks include pragma labeling, idiomatic OpenMP kernels, BLAS-level routines, and guided translation.

CRediT authorship contribution statement

The authors contributed across methodology, investigation, writing, conceptualization, and supervision, with distinct roles assigned to each contributor.

  • Strahinja Ljaljevic contributed methodology, investigation, and original-draft writing.
  • Josep Jorba contributed investigation and writing review and editing.
  • Sergio Iserte contributed conceptualization, investigation, writing review and editing, and supervision.

Funding

The authors report that this research received no specific grant funding from public, commercial, or not-for-profit funding agencies.

  • Funding: The authors received no specific grant from public, commercial, or not-for-profit funding agencies for this research.

Data availability

The study’s supporting datasets, figures, and bibliographic references are openly available in a dedicated GitHub repository. The repository also provides metadata for the 32 reviewed papers and the spreadsheets used for data collection and categorization.

  • Data availability: Supporting datasets, figures, and bibliographic references are openly available in a dedicated GitHub repository.Repository: https://github.com/sljaljevic/llm-hpc-survey.
  • Data availability: The repository includes metadata for the 32 reviewed research papers.
  • Data availability: The repository contains manuscript figures and spreadsheets used for data collection and categorization.Researchers are encouraged to reuse and extend these resources in future studies.

Declaration of generative AI and AI-assisted technologies in the writing process

An AI language model was used for language polishing, followed by thorough author review and editing. The authors retain full responsibility for the manuscript’s final content.

  • AI-assisted language polishing was thoroughly reviewed and edited by the authors, who take full responsibility for the final manuscript content.
Loading 2608.26110v1…