PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM Agents as Resilience Engineers for Scientific Applications

Hai Duc Nguyen, Tekin Bicer, Kyle Chard, Ian Foster, Bogdan Nicolae

arXiv:2610.11260v1cs.DC

TL;DR

Checkpoint/restart is essential for HPC resilience, but implementing it demands application-specific reasoning about recoverable state and consistent restart. This paper evaluates a no-human-in-the-loop agent pipeline on MPI applications and finds working implementations with low overhead and recovery efficiency comparable to human-written code, while fragmented state remains challenging.

  • Problem

    Checkpoint/restart implementation requires identifying recoverable state, selecting globally consistent points, and preserving application invariants during restart.

    Source: §II · p. 2

  • Method

    A no-human-in-the-loop generate–validate–revise pipeline evaluates LLM coding agents on 16 MPI applications with varied critical-state structures.

    Source: §I · p. 1, C. Validation Design · p. 4

  • Results

    Across 16 applications, the pipeline produced 41 working implementations with negligible failure-free overhead and recovery efficiency comparable to human-written code.

    Source: §I · pp. 1–2

  • Takeaways & Limitations

    Agent-driven resilience engineering is practical when critical state is visible or accessible through coherent abstractions.

    Source: §I · p. 1, §VII · p. 8

    Agents struggle with critical state that is modularized and fragmented across subsystems, particularly when validator feedback provides weak diagnosis.

    Source: §VII · p. 8

Abstract

from arXiv · show

Efficient checkpoint/restart support is essential for resilient HPC scientific applications, but implementing it requires substantial expertise: developers must identify recoverable state, choose globally consistent checkpoint points, and preserve application invariants during restart. We study whether frontier LLM coding agents can automate this process. We build a benchmark suite of 16 MPI applications spanning diverse domains, code sizes, and critical-state structures, and evaluate them with a no-human-in-the-loop generate--validate--revise pipeline for checkpoint/restart synthesis. Across the benchmark, the pipeline produces 41 working resilient implementations. Our results show that agent-driven resilience engineering is practical when critical state is visible or accessible through coherent abstractions: successful runs finish in under one hour on average, consume about 15M tokens, and produce implementations with negligible failure-free overhead and recovery efficiency comparable to human-written code. However, modularized and fragmented state remains a major limitation, with some failed attempts consuming over 100M tokens and 300 minutes without producing a working implementation.

I. INTRODUCTION

Checkpoint/restart requires application-specific expertise, motivating the question of whether coding agents can automate resilience engineering. A no-human-in-the-loop pipeline produced working implementations with low overhead and human-comparable recovery efficiency, but fragmented state remains difficult.

I. INTRODUCTION

  • Developers must identify recoverable state, choose globally consistent checkpoint points, and preserve communication and data-structure invariants during restart.These tasks require substantial engineering effort, application knowledge, and time.Source: §I · p. 1
  • Agents work when critical state is visible or accessible through coherent abstractions, but struggle with state fragmented across modules.The difficulty is inferring a complete and correct view of recovery state.Source: §I · p. 1
  • A four-class taxonomy captures how state visibility, mutability, abstraction, and fragmentation affect checkpoint/restart support.The classes are plain static, plain dynamic, encapsulated, and modularized.Source: §I · p. 1
  • A no-human-in-the-loop generate–validate pipeline drives LLM coding agents to add checkpoint/restart support to MPI applications.The pipeline produced 41 working implementations across 48 application–model settings.Source: §I · p. 1
  • For applications with directly visible critical state or coherent framework abstractions, agents reliably generated working checkpoint/restart implementations with negligible failure-free overhead and recovery efficiency comparable to human-engineered implementations in less than one hour, using about 15M tokens on average.Source: §I · pp. 1–2

II. BACKGROUND AND MOTIVATION

Checkpoint/restart limits progress lost to fail-stop failures, but implementing it efficiently requires application-specific knowledge and adaptation to changing deployment conditions. Coding agents may help with the engineering, though end-to-end resilience automation remains unestablished.

II. BACKGROUND AND MOTIVATION

  • Checkpoint/restart saves critical state to persistent storage and resumes from the latest checkpoint after a fail-stop failure, bounding rollback instead of losing all progress.Without explicit fault tolerance, fail-stop failures typically abort the entire tightly coupled MPI job.Source: §II · p. 2
  • Efficient checkpoint/restart requires identifying critical rather than reconstructable state and choosing safe points where distributed state is consistent.Correct restoration must preserve invariants across computation, communication, and I/O.Source: §II · p. 2
  • Checkpoint/restart efficiency varies with failure rate, checkpoint I/O overhead, platform, workload, and system load, so implementations may need repeated adaptation and validation.The Young/Daly formula relates optimal checkpoint period to failure rate and checkpoint I/O overhead.Source: §II · p. 2
  • Coding agents can inspect and edit repositories, execute tests, and use feedback iteratively, while checkpoint/restart has a known high-level workflow but application-specific low-level engineering.This motivates specification-driven synthesis and testing as a way to shift some implementation work from developers to agents.Source: §II · p. 2
  • Existing Agent-for-HPC studies address narrower tasks, leaving end-to-end checkpoint/restart automation and its distributed-state, restart, build, runtime, and overhead requirements unassessed.Passing tests alone may not establish correctness because automated repair systems can exploit weaknesses in testing.Source: §II · p. 2

III. RELATED WORK

Prior checkpoint/restart approaches trade developer effort against application-level efficiency, and no prior work had systematically studied frontier LLM agents for end-to-end automation. The paper also classifies applications by critical-state structure, which entails increasing checkpointing difficulty.

III. RELATED WORK

  • Mature checkpointing libraries reduce checkpoint size, runtime overhead, and storage cost, but still rely on developers to identify state, choose safe points, and implement restart logic.Other techniques optimize staging, flushing, aggregation, compression, and checkpoint cadence after those application-specific decisions are made.Source: §III · pp. 2–3
  • Transparent checkpointing reduces developer involvement by snapshotting whole processes, but can capture unnecessary state and face scalability, portability, and compiler-support constraints.Compiler-assisted approaches can struggle with dynamically allocated, templated, and irregular data structures.Source: §III · p. 3
  • Existing approaches tend to offer either low developer effort or application-level efficiency, and prior work had not systematically studied frontier LLM agents for end-to-end checkpoint/restart automation.Source: §III · p. 3

A. Critical State Classification

  • The four classes in Figure 1 progress from plain static through plain dynamic and encapsulated to modularized state, with increasing checkpoint/restart difficulty.Modularized state spans modules whose APIs and dependencies must be understood to reconstruct a complete recovery view.Source: A. Critical State Classification · p. 3, Figure 1 · p. 3
  • Plain static state remains fixed in application-managed primitives; plain dynamic state changes structure or ownership, while encapsulated state is exposed through framework APIs.The benchmark suite covers applications across domains, code sizes, computation and communication patterns, and critical-state structures.Source: A. Critical State Classification · p. 3

B. Resilient Code-Refactoring Pipeline

The resilient code-refactoring pipeline strips existing resilience logic, then has an agent iteratively generate checkpoint/restart code that an independent validator builds and tests. Validation feedback guides revisions.

B. Resilient Code-Refactoring Pipeline

  • Existing resilience logic, including framework checkpoint read/write hooks, is removed to create Vanilla while preserving computation and failure-free scientific output.The agent must infer the checkpoint/restart contract from intrinsic data structures and execution logic rather than reusing the original implementation.Source: B. Resilient Code-Refactoring Pipeline · p. 4
  • In each iteration, the agent inspects and modifies Vanilla to add checkpoint/restart while preserving its original functionality.An independent validator builds and tests the generated code after each generation attempt.Source: B. Resilient Code-Refactoring Pipeline · p. 4

C. Validation Design

The validator checks whether generated code preserves application behavior, recovers correctly after a failure, and implements genuine checkpoint/restart rather than exploiting the tests. Failures trigger structured feedback for revision.

C. Validation Design

  • Validation checks functionality, failure resilience, and trustworthiness.Functionality compares failure-free output with Vanilla; resilience tests recovery after killing an MPI rank; trustworthiness requires a genuine checkpoint/restart mechanism.Source: C. Validation Design · p. 4
  • A resilience test confirms a checkpoint was written before failure, post-restart output matches Vanilla, and recovery beats a full restart from scratch.The validator kills one MPI rank mid-run, relaunches the whole MPI run, and checks the recovery path.Source: C. Validation Design · p. 4
  • Runs last about 60–200 seconds, with failures injected uniformly between 50% and 80% of failure-free runtime.An upper bound on checkpoint size excludes whole-process memory dumps.Source: C. Validation Design · p. 4
  • Randomized inputs and a cleaned, sandboxed environment help prevent output replay, reuse of prior-test results, and copying reference solutions.These defenses address attempted validator gaming observed in the agents’ code-refactoring loops.Source: C. Validation Design · pp. 4–5

V. BENCHMARK SETUP

The benchmark evaluates whether coding agents can add checkpoint/restart without human intervention, how generated implementations compare with native code, and why attempts fail. It uses three LLM backends, a validation-capped generation loop, and measurements of generation cost and runtime efficiency.

V. BENCHMARK SETUP

  • The benchmark measures feasibility, efficiency against native human-written code, and challenges that explain failures.The authors make the benchmark suite available for reproducing the experiments.Source: §V · p. 5
  • The study uses Opus 4.7, Sonnet 4.6, and GPT-5.5 with OpenCode CLI and the VeloC checkpoint/restart library.Each application–model attempt is capped at 10 iterations; attempts that do not pass validation within the cap count as failures.Source: §V · p. 5
  • Each attempt records time, token and monetary costs, end-to-end execution time, and per-checkpoint footprint.Measurements use one development host, four MPI ranks per application, and three independent trials per configuration; the reported value is the mean.Source: §V · p. 5
  • Table II reports iterations to generate a working checkpoint/restart implementation as the median over successful attempts out of three trials, with per-application values giving the lowest count across models.The table marks cases where all three attempts failed with ×.Source: Table II · p. 5

A. Feasibility of Enabling Resilience

The pipeline produces working checkpoint/restart implementations across many applications, while fragmented state in modularized applications is a major obstacle. Agents typically inspect the code, identify state and consistent checkpoint points, integrate restart logic, and use validator feedback to revise their implementation.

A. Feasibility of Enabling Resilience

  • Across 16 applications and three LLM backends, the pipeline succeeds in 41 of 48 application–model settings.The result spans different domains, code sizes, and state structures without human intervention.Source: A. Feasibility of Enabling Resilience · p. 5
  • Agents trace the application from its entry point to the iterative computation, then use initialization to distinguish checkpoint state from reconstructable state.Variables updated inside the iterative region are selected for checkpointing, while input-derived or reconstructable variables are excluded.Source: A. Feasibility of Enabling Resilience · p. 5
  • Agents place checkpoints at consistent points, such as after MPI synchronization or output flushes, and put restart logic before compute kernels execute.They integrate through inline code, existing framework utilities, or a helper module, then use structured validator failures to propose targeted fixes.Source: A. Feasibility of Enabling Resilience · p. 6
  • The workflow is effective for most plain-static, plain-dynamic, and encapsulated applications, but fragmented state across modularized applications remains a limitation.Hidden fields and cross-module dependencies are difficult to infer, and validator feedback often reports symptoms rather than root causes.Source: A. Feasibility of Enabling Resilience · p. 6
  • Some modularized successes use a fallback of checkpointing all exposed module state, but this approach is inefficient and unreliable.It may exceed the iteration budget or fail to capture enough state for a particular application.Source: A. Feasibility of Enabling Resilience · p. 6

B. Code Generation Cost

Generation cost depends more on checkpointing difficulty and critical-state complexity than on codebase size. Models differ in time and token use: fewer iterations can reduce wall time, while less exploration can lower token costs but require more repair on difficult applications.

B. Code Generation Cost

  • 201.2 minutes for modularized successes versus 32 minutes for plain-static applications, a 6.3× increase from easiest to hardest group.Code generation alone rises from 14.1 minutes for plain-static applications to 141.9 minutes for modularized successes, reflecting the effort to reason through harder critical-state structures.Source: B. Code Generation Cost · p. 7
  • 58.3 minutes for Opus 4.7, 61.5 for GPT-5.5, and 100.4 for Sonnet 4.6 on average.Opus's advantage comes from fast completion: 11/14 successful runs pass validation with the fewest iterations; GPT-5.5 spends 4.5 minutes generating code per iteration, versus 9.9 for Opus and 12.8 for Sonnet.Source: B. Code Generation Cost · p. 7
  • Across applications ranging from 4K to 1M LoC, token usage remains mostly below 25M tokens, suggesting it does not grow directly with codebase size.Agents localize their reading and editing to likely computation, allocation, and integration sites rather than scanning the whole repository.Source: B. Code Generation Cost · p. 7
  • Plain-static applications average 5M tokens per attempt, while modularized applications consume 12.4× more tokens relative to that baseline.Input tokens account for 98.4%–99.7% of total usage; across models, average use is 22.8M for Opus 4.7, 30.4M for Sonnet 4.6, and 10.5M for GPT-5.5.Source: B. Code Generation Cost · p. 7
  • GPT-5.5 costs $53.7 per attempt, compared with $116 for Opus 4.7 and $93 for Sonnet 4.6.Its lower token use can make it cheaper, but less exploration can require extra repair iterations on complex applications.Source: B. Code Generation Cost · pp. 7–8

C. Efficiency of the Generated Resilience Code

Generated implementations combine negligible failure-free overhead with recovery efficiency comparable to human-written code, while checkpoint size varies with application state structure and model choices.

C. Efficiency of the Generated Resilience Code

  • Usually within 3%, LLM-generated implementations stay close to Vanilla in failure-free execution and are statistically indistinguishable from Reference.Under failure injection, they usually limit failure impact to single-digit seconds, matching or improving on Reference; Vanilla loses at least 50% of runtime.Source: C. Efficiency of the Generated… · p. 8
  • Plain-static and modularized applications have checkpoint sizes comparable to Reference, but modularized implementations dump the whole application state.For plain-static applications, agents select compact state; for modularized applications, both Reference and LLMs conservatively save the whole application state.Source: C. Efficiency of the Generated… · p. 8
  • On Athena++, Sonnet’s checkpoint is 53× larger than Reference’s, at 128 versus 2.4 MiB.On OpenLB, every LLM saves only the populations needed to resume the collision-streaming loop, while Reference dumps the full distribution-function lattice.Source: C. Efficiency of the Generated… · p. 8
  • SPARTA exposes coarse-grained state selection: LLMs save 96 B per particle and roughly 9–10 MiB per snapshot, versus Reference’s 7.8 MiB.The LLMs save the full Particle::OnePart struct, while Reference uses a restart-specific 80 B per particle Particle::OnePartRestart struct.Source: C. Efficiency of the Generated… · p. 8
  • Opus 4.7 stays within 1.3× of Reference when inflating checkpoint size and produces the smallest checkpoint among the three LLMs on 10 of 16 applications.Sonnet 4.6 has the Athena++ outlier; GPT-5.5 inflates footprint relative to Reference on most applications. Overall, LLM implementations usually achieve comparable storage overhead, but footprints can vary widely.Source: C. Efficiency of the Generated… · p. 8

VII. CONCLUSION

The paper finds agent-driven checkpoint/restart engineering feasible across a meaningful class of MPI applications, with success depending more on critical-state structure than codebase size.

VII. CONCLUSION

  • Across 16 applications, the no-human-in-the-loop pipeline produced 41 working implementations with negligible failure-free overhead and recovery efficiency comparable to human-written code.Agents were effective when recovery state was directly visible or accessible through coherent abstractions.Source: §VII · p. 8
  • Agents struggle when critical state is modularized, fragmented, and weakly diagnosed by validator feedback.The conclusion identifies critical-state structure, rather than codebase size, as the primary determinant of success.Source: §VII · p. 8
  • The paper points to resilience-aware agents combining code generation with state-dependency analysis, better source-code summarization, and more diagnostic validation feedback.Source: §VII · p. 8
Loading 2610.11260v1…