PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Agentic Runtime Healing

Zhensu Sun, Haotian Zhu, Bowen Xu, Xiaoning Du, Li Li, David Lo

arXiv:2408.01055v2cs.SEcs.AIcs.CR

TL;DR

Runtime errors cannot all be anticipated and handled in advance, while existing self-healing strategies rely on predefined heuristics. Healer uses an integrated LLM to generate and execute context-specific code that revises program state. Across benchmarks, GPT-4 proceeded with 73.0% of executions and helped 38.7% of programs produce correct results, achieving the highest PROCEED and CORRECT performance among evaluated LLMs.

  • Problem

    Because runtime errors and their locations are unknown in advance, existing self-healing systems rely on predefined heuristics, while anticipating and handling every possible error remains challenging.

    Source: §1 · pp. 1–2

  • Method

    Healer integrates an LLM into runtime error handling to generate and execute context-specific healing code that revises the program’s runtime state.

    Source: §2 · p. 3, §1 · p. 3

  • Results

    Across benchmarks, GPT-4 proceeded with 73.0% of executions and helped 38.7% of programs produce correct results, achieving the highest PROCEED and CORRECT performance among evaluated LLMs.

    Source: §3.1 · p. 6

  • Takeaways & Limitations

    The study provides a proof of concept that LLM-driven runtime error handling is feasible.

    Source: §6 · p. 10

    Without further constraints, Healer’s recovery is not guaranteed to be correct or safe because generated code can exhibit risky behaviors.

    Source: §4.1 · p. 9

Abstract

from arXiv · show

Self-healing systems have long been a focus of research, aiming to enable software to recover from unexpected runtime errors without human intervention. Traditional approaches rely on predefined heuristic rules, such as reusing error handlers or rolling back to checkpoints, but these methods struggle to adapt to the diverse range of runtime errors. The emergence of Large Language Models offers a new opportunity to address this challenge. Leveraging their ability to understand and generate code and natural language, we propose using LLMs to dynamically generate error-handling strategies in real time, tailored to specific runtime contexts such as error messages and program states. We demonstrate the feasibility of this approach by designing such a framework, Healer, and empirically showing that it can handle runtime errors with a high success rate. When an unanticipated runtime error occurs, Healer leverages its internal LLM to generate bespoke error-handling code. The generated healing code is then executed to produce a corrected program state, allowing the program to continue execution with minimal disruption. We evaluate Healer across four code datasets and three state-of-the-art LLMs (GPT-3.5, GPT-4, and CodeQwen-7B), where GPT-4 can successfully recover from 72.8% of runtime errors, underscoring the promise of LLMs in this domain. Despite these promising results, challenges remain, particularly regarding the trustworthiness of LLM-generated code and its integration into existing systems. We mention potential solutions, such as safety checks and Healer-aware programming, to mitigate risks and ensure reliable operation. This work represents the first step toward agentic runtime healing, paving the way for more adaptive, resilient, and self-healing software systems.

1 Introduction

Healer uses LLMs to generate runtime-specific healing code and dynamically recover from unanticipated errors. In the evaluation, GPT-4 continued execution in 72.8% of instances and produced correct results in 39.6%.

1 Introduction

  • Because unanticipated errors and their locations are unknown beforehand, predefined strategies such as restarts and rollbacks adapt poorly to diverse errors and are difficult to implement.Identifying every possible runtime error and providing handlers remains challenging and often impractical; an industry study also emphasizes the need for more adaptive self-healing approaches [38].Source: §1 · pp. 1–2
  • Healer prompts an LLM with runtime context to generate and dynamically execute healing code that revises faulty state rather than patching the original program.The intended outcome is to avoid termination and let execution continue meaningfully with a corrected state.Source: §1 · pp. 2–3
  • In the SOS example, a race condition makes directory creation crash and lose collected logs; Healer renames the output directory so execution can continue.The healing snippet is dynamically executed to produce an updated program state.Source: §1 · pp. 2–3
  • Without fine-tuning, GPT-4 continued execution in 72.8% of instances and produced correct results in 39.6%.The evaluation covered 1,185 (code, test case) pairs across four datasets and tested GPT-3.5, GPT-4, and CodeQwen. Fine-tuning improved GPT-3.5 and CodeQwen interruption-free instances by 31.7% and 23.1%, respectively, and correct-result instances by 26.8% and 65.8%, respectively.Source: §1 · p. 3
  • Executing LLM-generated code raises trustworthiness risks, including ignoring key original logic; predefined safe functions are one proposed way to constrain healing.The paper identifies safety checks and Healer-aware programming as potential mitigations.Source: §1 · p. 3

2 Proposed Framework

Healer integrates a local or remote LLM into runtime error handling, using error context to generate code that can correct the current program state and resume execution.

2 Proposed Framework

  • For an unhandled runtime error, Healer prompts an integrated LLM with program context, executes generated healing code, and resumes at the next line if execution succeeds.If the healing code raises another error, the program terminates and reports it. The workflow assumes an interpreted language whose runtime persists state between execution steps.Source: §2 · pp. 3–4, Figure 2 · p. 4
  • Successful healing updates state only for the current session; it does not permanently fix the original program, so the underlying bug still requires reproduction and development-time repair.The framework describes the healing runtime for interpreted languages such as Python and JavaScript.Source: §2 · pp. 3–4
  • Its collector supplies source code, error message, location, and pre-error state; a structured prompt adds task instructions, goals, constraints, and an example, marking the error block with tags.The error location is highlighted in the source rather than presented as a standalone prompt field.Source: §2 · p. 4
  • Healing code runs against the pre-error state without rerunning the original program; execution may use the same runtime or an isolated session, sandbox, or VM for security.The runtime that executes the healing code is called the healing runtime.Source: §2 · pp. 4–5

3 Evaluation of Healer

Healer was evaluated by executing Python code with generated healing code, and the results show that LLM-based healing can improve runtime-error outcomes. Fine-tuning further improves performance on CodeNet.

3 Evaluation of Healer

  • The evaluation used a proof-of-concept Python executor and tested runtime-error instances from CodeNet, HumanEval+, MBPP+, LiveCodeBench, and DebugBench with GPT-3.5, GPT-4, and CodeQwen.The evaluated instances included human-written code, LLM-generated code, and code with implanted bugs.Source: §3 · p. 5, §3.1 · p. 5

3.1 LLMs Can Effectively Handle Runtime Errors

  • GPT-4, GPT-3.5, and CodeQwen proceeded in 73.0%, 48.3%, and 35.1% of executions and produced correct results in 38.7%, 23.5%, and 13.0%, respectively.GPT-4 had the highest CORRECT and PROCEED metrics across benchmarks.Source: Table 1 · p. 6, §3.1 · p. 6
  • On DebugBench, GPT-4 completed 338 instances (77.5% of errors) and corrected faulty states in at least 177 (40.6%); on HumanEval and MBPP, it corrected 11 and 1 additional instances.Healer also achieved comparable performance on LiveCodeBench, which the authors use to assess potential data-leakage effects.Source: §3.1 · p. 6
  • Traditional Retry or Rollback would proceed in 0% of these deterministic-error cases, while an error steamroller achieved only 4.3% CORRECT.The authors distinguish crash avoidance from functional recovery: a steamroller may let execution continue but cannot correct program state.Source: §3.1 · pp. 6–7

3.2 Fine-Tuning Further Improves Healer’s Effectiveness

  • Fine-tuning increased GPT-3.5 CORRECT instances by 26.8% and PROCEED instances by 31.7%, while CodeQwen increased by 65.8% and 23.1%, respectively.Fine-tuned GPT-3.5 produced 90 CORRECT instances, compared with GPT-4’s 96 on CodeNet. The 586-pair instruction-tuning dataset used samples from all benchmarks except CodeNet, which was reserved for testing.Source: §3.2 · p. 7, Table 2 · p. 7

4 Remaining Challenges

Healer autonomously executes LLM-generated code, which can introduce unintended behavior and risks. The section examines these behaviors and outlines safeguards for more reliable use.

4.1 LLMs can introduce unexpected behaviors

  • Two experienced Python developers jointly examined 204 incorrect healing-code samples that proceeded execution but produced incorrect results; non-executable code was excluded, and the analysis was not intended as a comprehensive taxonomy.The goal was to empirically demonstrate behaviors Healer might introduce during healing.Source: §4.1 · pp. 8–9
  • Among investigated samples, 75% exhibited recognizable behaviors, including importing new libraries (27.3%), changing variable values (21.0%), initializing undefined variables (10.2%), and adjusting types (5.9%).The authors identified 19 distinct behaviors; omitting context (2.4%) or return statements (2.9%) can disrupt expected data flow.Source: §4.1 · p. 9

4.2 A Trustworthy Future?

  • Without constraints, Healer’s recovery is not guaranteed safe or correct, but recognizable behaviors can be checked against a whitelist to restrict interventions to those acceptable for the guarded program.Static checks could detect behaviors such as omissions; other possible safeguards include dynamic testing, static analysis, and verification. Prior work also raises concerns about executing generated code [39], including LExecutor [33].Source: §4 · p. 8, §4.1 · p. 9, §4.2 · p. 9
  • Healer-aware systems can accommodate interventions with output warnings, detailed error logs, and a “comfort zone” in non-critical components; a predefined LLM toolkit can further make behavior predictable.The toolkit could restrict the LLM to predefined functions, such as a data-saving function, following practices in AI agents for software development [35].Source: §4.2 · pp. 9–10

5 Discussion

The discussion outlines Healer’s deployment prerequisites and real-world integration challenges, and explains how its framework could extend beyond Python despite added complexity for compiled languages.

Infrastructure and Deployment Implications

  • Healer requires runtime-state access, an execution environment for generated healing code, and an LLM service, whose remote and local deployment options have different constraints.Remote APIs need stable network connectivity; local models need sufficient computational power, with compression [31] and quantization [20] among techniques for enabling inference on user devices.Source: §5 · p. 10

Healer in Real-world Systems

  • Real-world use requires targeted retrieval of error-relevant code and integration with existing recovery mechanisms, including rollback as a fallback if Healer fails.Reading an entire, potentially million-line codebase for each error is inefficient; repository-level bug localization and code completion [40] offer relevant retrieval methods.Source: §5 · p. 10

Extension to other Programming Languages

  • Although implemented in Python, Healer’s framework is language-agnostic; interpreted languages fit naturally, while compiled languages require runtime compilation or dynamic loading.Java’s Compiler API and C++ shared-library loading can address compiled-language execution complexity, though extension requires engineering effort.Source: §5 · p. 10

6 Conclusion

The paper introduces AI-driven runtime error handling as a pathway toward self-healing software and presents Healer as a proof of concept, experimentally showing a high success rate. It also identifies LLM uncertainty and compatibility with existing architectures as challenges.

  • Healer provides a proof of concept for AI-driven runtime error handling, which the authors propose as a new pathway toward self-healing systems.The authors describe this as the first proposal of the idea.Source: §6 · p. 10
  • Significant challenges remain, particularly uncertainty in LLM behavior and compatibility with existing software architectures.Source: §6 · p. 10
  • The Healer artifact is available at https://github.com/v587su/Healer.Source: §6 · p. 11
Loading 2408.01055v2…