PaperlayerView exact revision on arXiv

Source-linked AI summary

Trustworthy Runtime Error Healing in Real-World Repositories: A Benchmark and Guardrail

Gou Tan, Pengfei Chen, Zhensu Sun, Jieke Shi, Junkai Chen, Ting Zhang, Weifeng Sun, Junda He, Shuai Liang, Chuanfu Zhang, Lwin Khin Shar, David Lo

arXiv:2609.39086v1cs.SEcs.AI

TL;DR

Runtime healing research has evaluated LLM-generated recovery mainly on small programs, while repository-level context and the safety effects of running healing code in a live process remain concerns. This paper introduces HealBench and HealGuard to evaluate recovery and check whether healing changes reach protected operations. The best reported setting completes 38.11% of instances and passes 28.68% of target tests, while controlled tests show HealGuard detects all unsafe cases.

  • Problem

    Runtime healing has been evaluated on small competition programs, while repository-level healing requires cross-file context and its live-process effects may reach protected operations.

    Source: §1 · p. 2

  • Method

    The paper builds HealBench for repository-level healing and introduces HealGuard, which uses static and dynamic taint checks on healing code written in HealCore.

    Source: §1 · p. 2

  • Results

    38.11% of instances reach completion and 28.68% pass the target test with Codex and GPT-5.6-Terra; HealGuard detects all 342 unsafe controlled cases with 59.38% precision.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The study connects repository-level runtime-healing evaluation with checks for whether healing-changed state reaches developer-protected operations.

    Source: §6 · p. 10

    HealGuard covers only declared protected operations and explicit data flow, has a high false positive rate, and does not eliminate risks from running healing code in a live process.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Runtime error healing lets a crashed program continue by generating code that repairs its live runtime state. Recent work shows that LLMs can generate such healing code, but it is evaluated only on small competition programs, and executing LLM-generated code inside a live process raises safety concerns that remain unaddressed. In this paper, we take LLM-based runtime healing toward practical use in real-world repositories. We first build HealBench, a benchmark of 265 runtime errors from 18 real-world repositories, each paired with a reference execution on the patched version. HealBench also provides a unified framework that lets LLM agents heal with cross-file context and live runtime state. We then design HealGuard, which requires healing code to be written in HealCore, an analyzable subset of Python, and uses static and dynamic taint analysis to check whether state changed by healing reaches operations protected by developers. We evaluate a dedicated healing method and three general coding agents with three backbone LLMs. The best setting resumes execution in 38.11% of instances and passes the target test in 28.68%, showing that existing agents can already heal a meaningful share of real repository-level crashes. However, among executions that pass, HealGuard flags 17.4% whose healing-changed state may reach a protected operation. On 684 controlled cases, HealGuard detects all unsafe cases, at the cost of a 68.42% false positive rate.

1 INTRODUCTION

The paper addresses the limits of runtime healing in small programs and the safety risks of running generated code in live processes. It introduces a repository-level benchmark and a guardrail, then evaluates agents’ recovery and safety outcomes.

1 INTRODUCTION

  • Prior healing evaluations use small competition programs, while repository-level healing needs cross-file context and may involve successive crashes and effects on protected operations.A Pillow setting that allows truncated images to load can also cause later integrity checks to be skipped.Source: §1 · p. 2
  • HealBench contains 265 runtime errors from 18 repositories, pairs each with a patched-version reference execution, and lets agents inspect repositories and live state through a unified framework.The protocol supports both dedicated healing methods and general coding agents.Source: §1 · p. 2
  • HealGuard treats values computed and locations modified by healing as taint sources and checks whether they reach developer-protected operations.It uses HealCore, a restricted Python subset, and combines static potential-flow analysis with checks based on runtime events.Source: §1 · p. 2
  • 38.11% of instances reach completion and 28.68% pass the target test with Codex and GPT-5.6-Terra; HealGuard flags 17.4% of passing executions.On 684 controlled cases, HealGuard detects all 342 unsafe cases, with 59.38% precision.Source: §1 · p. 2

2 RELATED WORK

Prior work covers runtime recovery, coding agents, and program analysis, but repository-level healing requires combining cross-file context with live runtime state.

Runtime Error Healing

  • Runtime healing includes LLM-generated in-process recovery, developer-defined actions for failed training jobs, development-time exception handling, and pre-execution error detection.Repository-level healing requiring cross-file context remains insufficiently explored.Source: §2 · p. 3

LLM-Based Coding Agents

  • Coding agents interact with repositories through file reading, code search, shell commands, and editing, but runtime healing also requires understanding and modifying live state.The cited agent tasks focus primarily on source code.Source: §2 · p. 3

Program Analysis for Runtime Safety

  • The paper applies source-to-sink taint analysis to check whether state changes from generated healing code propagate to later security-sensitive operations.The approach combines static analysis of potential paths with dynamic analysis during execution.Source: §2 · p. 3

3 HEALBENCH: EVALUATING REPOSITORY-LEVEL RUNTIME ERROR HEALING

HealBench pairs real-repository runtime errors with patched reference executions and provides a shared framework for agents to inspect and heal crashes. Its dataset and framework cover multi-file executions, live state, and healing attempts that can be revised or abandoned.

3 HEALBENCH: EVALUATING REPOSITORY-LEVEL RUNTIME ERROR HEALING

  • HealBench reproduces crashes, runs healing code at the crash point, and continues execution to evaluate recovery against patched-version runs.The framework separates execution and healing environments and uses a unified interaction protocol.Source: §3 · p. 3, §3.2 · p. 4

3.1 DATASET CONSTRUCTION

  • 265 runtime errors remain after filtering 22,341 candidates; patched runs span a median of 37 files and 130 functions.The 265 instances cover 18 repositories; TypeError, AttributeError, and ValueError account for 81.1%.Source: Figure 2 · p. 4, §3.1 · p. 4
  • HealBench selects executable repository tasks with reference patches, retains tests that fail on buggy versions and pass on patched versions, and manually reviews suitable errors.Errors must occur during repository execution, stem from the target bug, and allow execution to resume in the current process.Source: §3.1 · p. 4

3.2 EVALUATION FRAMEWORK

  • The execution side instruments repository statements to intercept runtime errors, pauses, and sends the exception, traceback, local variables, crash statement, and enclosing function to the healing side.Healing code runs in a temporary environment sharing mutable objects with the paused execution; failed attempts do not automatically revert shared-object changes or external effects.Source: §3.2 · pp. 4–5
  • Agents inspect a read-only repository copy and probe runtime values, then submit or revise healing code; they may stop and let the original exception propagate.Probe changes to mutable objects and external state are not automatically reverted.Source: §3.2 · p. 5

4 HEALGUARD: A GUARDRAIL FOR UNSAFE ERROR HEALING

HealGuard treats healing-generated values and state changes as untrusted, checking whether they reach developer-protected operations during healing or later execution. It restricts healing code to HealCore and combines static analysis with dynamic checking when static analysis is inconclusive.

4 HEALGUARD: A GUARDRAIL FOR UNSAFE ERROR HEALING

  • HealGuard treats direct operations and later effects of healing as potentially unsafe when healing-introduced values reach developer-protected operations.Passing the target test does not rule out these effects.Source: §4 · p. 5

4.1 TAINT SPECIFICATION

  • Developers specify protected operations and their key inputs; healing-written locations and computed values become taint sources, while protected-operation inputs are sinks.Taint propagates through explicit data flow, and safe healing requires every sink invocation to receive an untainted key input.Source: §4.1 · pp. 5–6

4.2 HEALCORE: KEEPING HEALING CODE ANALYZABLE

  • HealCore excludes constructs that obscure behavior, including dynamic execution, imports outside an allowlist, function definitions, and interpreter-internal access.A syntax-tree gate rejects healing code that does not belong to HealCore before execution.Source: §4.2 · p. 6

4.3 TAINT CHECKING

  • For unknown static results, HealGuard resumes with dynamic checking; wrappers block a protected operation when its key input may carry monitored taint.For a tainted static result, HealGuard raises the original exception, but this does not undo changes already made by healing code.Source: §4.3 · pp. 6–7
  • Static analysis labels a run tainted when a healing source reaches a reachable protected input, clean when no path exists and analysis is complete, and unknown otherwise.Unmodeled constructs such as reflection, container aliasing, or native calls can make analysis incomplete.Source: §4.3 · p. 6

5 EVALUATION

The evaluation compares a dedicated healer and general coding agents on repository-level healing, then tests HealGuard on controlled unsafe and safe cases. Results show that continuing execution does not guarantee test success, while HealGuard detects unsafe cases at a substantial false-positive cost.

5 EVALUATION

  • Evaluation compares Healer and three coding agents across three LLMs, measuring trace similarity, confirmed completion, and target-test success, plus 342 unsafe and 342 safe controlled cases.The runtime-healing evaluation uses HealBench instances; unsafe controlled cases contain a heal-source dependency into a protected-operation input, while safe cases do not.Source: §5 · p. 7

5.1 RESULTS OF RUNTIME ERROR HEALING

  • 5.12%–38.11% Proceed Rate and 3.54%–28.68% Correct Rate show limited performance, with correctness below continuation in every setting.For Vanilla runs, Trace Similarity ranges from 47.90% to 75.35%, but matching more execution points does not establish an expected result.Source: §5.1 · p. 7, Table 2 · p. 8
  • No method leads under every backbone: mini-SWE-agent has the highest Correct Rate with GLM-5.2, while Codex leads with GPT-5.6-Terra.GPT-5.6-Terra generally yields the highest Correct Rate for a given method, but mini-SWE-agent without HealGuard reaches 25.28% with GLM-5.2 versus 24.53% with GPT-5.6-Terra.Source: §5.1 · p. 8
  • With GPT-5.6-Terra, Healer mostly performs Heal, OpenHands uses more Explore and Inspect State, and Codex has the highest Reraise share.Healer has no separate Explore or Inspect State actions; Reraise can reflect invalid responses or exhausted budgets, not just deliberate refusal.Source: §5.1 · p. 8
  • Recorded crash-count medians are 6 for Healer, 7 for mini-SWE-agent, 5 for OpenHands, and 4 for Codex.These counts reflect intercepted errors, retries, healing state changes, and logging coverage, not all program exceptions; earlier healing can affect later crashes.Source: §5.1 · p. 8

5.2 RESULTS OF UNSAFE HEALING DETECTION

  • HealGuard blocks all 342 unsafe controlled cases (100% recall), but also blocks 234 of 342 safe cases (68.42% false positive rate).Static analysis blocks 177 unsafe and 12 safe cases; dynamic checking of the 495 cases it does not block catches the remaining 165 unsafe cases and blocks 222 safe cases. In the Figure 6 case, static analysis returns unknown for an external API, and dynamic checking blocks because independence from healing-changed state cannot be established.Source: §5.2 · p. 9, Figure 6 · p. 9

6 CONCLUSION

The paper connects repository-level healing evaluation with safety checks, then details how healing resumes execution and how HealGuard checks healing-induced dependencies on protected operations.

6 CONCLUSION

  • The study connects repository-level runtime-healing evaluation with checks designed to detect and block unsafe healing.It presents this as a basis for building more reliable and safe self-healing systems.Source: §6 · p. 10

A.1 RUNTIME ERROR HEALING

  • After successful healing, execution continues after the injected handler, while later errors trigger new interactions using the current runtime state.Continuation does not automatically retry the failed statement, and a compound statement may have unfinished work skipped.Source: §A.1 · p. 14
  • Cross-file context helps the agent locate relevant state: in the matplotlib example, it rebinds a SubFigure to its parent Figure to access the renderer.The figure marks the runtime error in red and healing code in green.Source: §A.1 · p. 14, Figure 8 · p. 15

A.2 HEALGUARD WORKFLOW

  • Developers declare protected operations and each operation’s key input, which HealGuard treats as a sink, including inputs that determine external effects or security checks.For a security check, the declared key input is the condition deciding whether the check runs.Source: §A.2 · p. 15
  • HealCore restricts healing code’s syntax and capabilities, but the policy covers explicit dependencies on declared protected inputs, not complete correctness or every side effect.A policy violation identifies a prohibited dependency, not necessarily malicious behavior.Source: §A.2 · pp. 15–16
  • HealGuard labels healing-computed values and changed locations, then checks static data-flow paths from those sources to reachable protected-operation inputs.The implementation distinguishes tainted, clean, and unknown analysis outcomes.Source: §A.2 · p. 15
  • When static analysis is unknown, execution resumes with dynamic monitoring; a nonempty label estimate at a protected operation causes the wrapper to block it.The dynamic estimate can conservatively assign all monitored labels when an input cannot be established as an independent literal.Source: §A.2 · p. 16

A.3 IMPLEMENTATION AND SAFETY BOUNDARIES

The appendix distinguishes HealGuard’s intended workflow from implementation coverage and describes how HealBench candidates are collected, filtered, and reviewed.

A.3 IMPLEMENTATION AND SAFETY BOUNDARIES

  • The implementation approximates healing sources by mapping recorded changed names to graph nodes, and successful-healing records omit failed submissions and probes that may still change state.Mappings use file, function, and line information with nearest-node fallbacks.Source: §A.3 · p. 16
  • Implementation checks do not establish complete enforcement: project calls may have unanalyzed effects, mappings may remain unresolved, and Sink wrappers record calls without establishing that they were blocked.The predictions do not establish the required enforcement conditions for historical runs.Source: §A.3 · pp. 16–17

B.1 ERROR COLLECTION

  • HealBench candidates come from executable tasks with repository versions, environments, tests, and reference patches; buggy and patched runs link each retained error to a target test and reference execution.The patched test must pass, and its relation to the target bug is checked during filtering and review.Source: §B.1 · p. 17

B.2 ERROR FILTERING AND MANUAL REVIEW

  • Filtering retains errors caused by target bugs in repository code when the process remains available for healing, excluding cases such as environment failures and expected exceptions.The stated rules describe this benchmark’s studied setting, not failures that are impossible to heal in every environment.Source: §B.2 · p. 17
  • Candidates are deduplicated using source-instance identity or the test-name and error-message pair, then manually reviewed for bug relation, location, and suitability.Review uses the target test, traceback, crash context, and reference patch.Source: §B.2 · p. 17

B.3 DATASET COMPOSITION

HealBench contains repository-level runtime errors, while the appendix describes the dataset’s composition and the methods, execution settings, and metrics used for evaluation.

B.3 DATASET COMPOSITION

  • HealBench contains 265 instances from 18 repositories across 15 error types, with most instances from R2E-Gym-Lite.The dataset includes 217 R2E-Gym-Lite, 32 SWE-Bench-Full-Test, 10 SWE-Bench-Pro, and 6 SWE-Bench-Verified instances.Source: §B.3 · p. 17
  • TypeError, AttributeError, and ValueError are the most frequent error types, while pandas and numpy contribute the largest repository groups.Results reflect this dataset composition rather than the natural distribution of runtime errors across repositories.Source: §B.3 · pp. 17–18
  • Pre-crash executions have medians of 37 files and 130 functions, but these counts do not measure the cross-file context needed for each healing task.The reported statistics describe code traversed before the recorded crash.Source: §B.3 · p. 18
  • Target tests assess continued-execution behavior, while patched reference paths support location comparison; path overlap does not establish semantic equivalence or the full program contract.Healing need not reproduce the patched execution’s exact runtime state.Source: §B.3 · p. 18

C.1 HEALING METHODS AND BACKBONE LLMS

  • The evaluation compares Healer, mini-SWE-agent, OpenHands, and Codex with three backbone LLMs on the same HealBench instances.The agent interfaces differ, and the methods’ adapters are part of the comparison.Source: §C.1 · pp. 18–19

C.2 EXECUTION ENVIRONMENT AND CONFIGURATION

  • Repository execution uses fresh Docker containers, but other filesystem locations may remain writable and network access is not explicitly disabled.Hardware, image digests, and resolved dependencies require historical configuration records.Source: §C.2 · p. 19
  • The configured interaction budget uses adapter-specific counting, and the 3,180 method-model-instance records are not repeated trials across random seeds.Effective Codex generation settings remain unverified because the shown call does not explicitly set temperature.Source: §C.2 · p. 19

C.3 HEALING EVALUATION

  • Proceed rate and correct rate assess target-call completion and target-test success separately, while trace similarity compares counted execution locations with patched references.The trace calculation retains repeated visits but ignores global order, and a score of 1 does not establish correct output, state equivalence, or safety.Source: §C.3 · p. 20

C.4 HEALGUARD EVALUATION

The evaluation combines controlled HealGuard checks with repository-level healing outcomes and supplements them with analyses of checking, runtime overhead, and model cost. Its comparisons have important limits: the simulated labels are provisional, selected-case review does not establish independent validation, and the reported timing does not measure the full HealGuard process.

C.4 HEALGUARD EVALUATION

  • The balanced evaluation set contains 342 positive and 342 negative configurations paired around 62 healing points, with provisional labels and no assessment of global-change coverage or the HealCore gate.It uses deterministic templates, synthetic Sink events, and local AST graphs rather than application replay with production CodeQL inputs.Source: §C.4 · p. 21
  • Selected-case results are compared with supplied review labels, but the paired Sink-insertion prompt is not proof of independent review.Documented sampling, blinding, adjudication, and reviewer agreement are not established by the available review description.Source: §C.4 · p. 21
  • All 495 configurations passed to dynamic analysis were recorded as known and proven unreachable, so the simulation does not validate fallback restricted to unknown static verdicts.Independent checks use the full set, while dynamic analysis uses the static-retained subset.Source: §C.4 · p. 21
  • The evaluation defines TP, FN, FP, and TN by positive-case and negative-case predictions, but prediction accuracy does not establish that protected operations were prevented during execution.Simulated labels and selected-case review labels are evaluated separately.Source: §C.4 · p. 21

C.5 RUNTIME OVERHEAD AND MODEL COST

  • The reported timing measures function instrumentation rather than the full HealGuard process, and a complete evaluation would require matched workloads and separate measurements of analysis stages.The available timing does not measure the complete method.Source: §C.5 · p. 22
  • Estimated model costs use complete usage records, exclude cache discounts, infrastructure, and human effort, and do not measure added application latency.Partial and missing usage are kept separate, and configured prices were not verified against contemporaneous bills.Source: §C.5 · p. 22

D.1.1 HEALING WITH HEALGUARD

  • With HealGuard, Healer using GPT-5.6-Terra retained 167 of 265 executions at a 15.57% correct rate, versus 16.98% before selection.These percentages describe different retained sets; the comparison also excludes incomplete analyses, so some OpenHands baselines use fewer than 265 executions.Source: Table 5 · p. 22, §D.1.1 · pp. 22–23

D.1.2 HEALING WITHOUT HEALGUARD

  • With GPT-5.6-Terra, Codex had the highest proceed and correct rates, while mini-SWE-agent had the highest trace similarity.Completing the target call, passing the test, and matching the reference path are different outcomes; comparisons do not isolate cross-file context or runtime-state inspection.Source: Table 6 · p. 23, §D.1.2 · p. 23, §D.2 · p. 23

D.2.1 ERROR TYPES AND REPOSITORIES

Healing correctness varies across error types and repositories. Comparisons by category and repository therefore qualify any overall performance assessment.

D.2.1 ERROR TYPES AND REPOSITORIES

  • With GPT-5.6-Terra, Codex exceeds Healer on TypeError, AttributeError, and ValueError, but correct rates remain below 33% in these common categories.High rates in rare categories have small denominators and provide less evidence of a consistent advantage.Source: §D.2.1 · p. 23, Table 7 · p. 24
  • Codex with GPT-5.6-Terra reaches 80.00% correct on scrapy, 17.86% on pandas, and 11.25% on numpy.The larger pandas and numpy groups contribute more to the overall rate, while high rates in small repositories provide limited evidence for generalization.Source: §D.2.1 · pp. 23, 25, Table 8 · p. 24

D.2.2 AGENT INTERACTION AND HEALING COST

Without HealGuard, agent actions, recorded crash counts, and model usage vary across agents and backbone LLMs. The reported crash and cost measures have limits that constrain their interpretation.

D.2.2 AGENT INTERACTION AND HEALING COST

  • Action shares vary by backbone: mini-SWE-agent mainly submits healing code with GPT-5.6-Terra but inspects runtime state with GLM-5.2, while repository exploration dominates several other settings.Healer has no separate exploration or inspection tools; action shares pool trace counts, and “Reraise” includes invalid responses, exhausted budgets, and adapter failures.Source: Figure 9 · p. 25, §D.2.2 · p. 25
  • Healer with DeepSeek-V4-Flash records a median of 18 crashes per execution, compared with 2 to 3 for the other agents using that model.Counts depend on interception, retries, budgets, logging, and early termination; fewer recorded crashes do not establish easier healing.Source: Figure 10 · p. 25, §D.2.2 · p. 25
  • Healer generates the fewest mean output tokens across models, while the agent with the most output varies by backbone; more output does not consistently correspond to a higher correct rate.With DeepSeek-V4-Flash, OpenHands generates more output tokens than Healer but passes fewer target tests; mean cost uses complete usage records with differing coverage.Source: §D.2.2 · p. 25

D.3 HEALGUARD ANALYSIS

  • HealGuard comparisons use saved decisions rather than measured runtime intervention.The comparisons cover simulated decision accuracy, agreement with selected-case labels, and healing outcomes among retained executions.Source: §D.3 · p. 26

D.3.1 DECISION ACCURACY AND ERROR PATTERNS

The evaluation reports strong detection of labeled positive cases alongside substantial false positives, and describes distinct sources of error in static and dynamic analysis.

D.3.1 DECISION ACCURACY AND ERROR PATTERNS

  • On 495 cases retained by static analysis, dynamic taint analysis reaches 100.00% recall and 42.64% precision, with a 67.27% false positive rate.In the sequential setting, dynamic analysis checks every case retained by static analysis; these simulated cases were known and proven unreachable, not unknown.Source: §D.3.1 · p. 26, Table 9 · p. 26
  • On 30 selected cases and 265 mini-SWE-agent cases with GPT-5.6-Terra, dynamic analysis detects all labeled positive cases but increases false positives relative to static analysis on the latter group.The supplied passages do not report the validation metrics’ values.Source: §D.3.1 · p. 26, Table 10 · p. 26, Table 11 · p. 26
  • Static-analysis errors in the simulation include missed function-call and object-access dependencies, plus flags on inactive branches and unrelated fields.These errors concern simulation graphs, not CodeQL analyses in general.Source: §D.3.1 · p. 26
  • Dynamic analysis conservatively flags some independent inputs when their independence cannot be established, so these false positives do not confirm actual state propagation.The uncertainty can involve variables and transformed expressions.Source: §D.3.1 · p. 26

D.3.2 COMPONENT ANALYSIS

Component checks reject executions but test outcomes alone cannot establish whether rejections are false positives or confirm unsafe healing; function instrumentation adds 1.82% median execution time. Case studies distinguish extra effects from correction and show that correct healing may preserve process scope or reject invalid input.

D.3.2 COMPONENT ANALYSIS

  • 360 executions are rejected by static analysis and 317 by dynamic analysis, but test outcomes alone cannot identify false positives or confirm unsafe healing.Each analysis retains 450 test-successful executions and removes 95; proceeding and correct rates use allowed executions as the denominator. The comparisons do not evaluate the HealCore gate or replace component ablations.Source: §D.3.2 · p. 27

D.3.3 RUNTIME OVERHEAD

  • A 1.82% relative increase in median execution time accompanies function instrumentation, from 2.64127 ms to 2.68943 ms.The measurement uses 8,000 repetitions of the same workload and excludes source recording and taint analysis.Source: §D.3.3 · p. 27

D.4 CASE STUDIES

  • Target-test success does not establish the absence of extra side effects, and whether an operation violates policy depends on which operations are protected.The case studies compare healing code with reference patches and distinguish side effects from healing correctness.Source: §D.4.1 · p. 27, §D.4 · p. 27

D.4.1 HEALING SAFETY CASES

  • In qutebrowser, Healer deletes a cache file that the reference patch preserves, while a Pillow healing attempt invokes subprocess execution to run pip install olefile.The cache-preservation check confirms deletion; the recorded subprocess call fails, so it shows execution rather than successful installation.Source: §D.4.1 · p. 27, Figure 11 · p. 28
  • Cookie healing prints names and values unnecessarily; this can expose session data, but the example does not establish transmission to an external recipient.The reference patch iterates over cookie objects directly without adding output.Source: §D.4.1 · pp. 27–28, Figure 12 · p. 28

D.4.2 HEALING CORRECTNESS CASES

  • For a missing function, correct healing assigns a local value instead of adding a process-wide function; for invalid input, it preserves rejection by raising ValueError.The paired examples contrast local repair with a builtins change and preserving the required input check with returning float(s) while skipping it.Source: §D.4.2 · p. 28, Figure 13 · p. 28

E LIMITATIONS AND FUTURE WORK

This section describes limitations in HealBench and HealGuard’s analysis coverage, runtime execution, evaluation scope, feedback, and response handling, including caveats around correctness and validation. It also outlines future work on improving coverage and containment and validating recovery and safety through broader, independently labeled evaluations.

E LIMITATIONS AND FUTURE WORK

  • HealGuard’s analysis can miss dependencies because of incomplete source, sink, and CodeQL call-model coverage, while its explicit-flow policy omits general control dependence and conservative rejection can block independent inputs.These limitations can hide state changes or dependencies and can cause false rejections.Source: E LIMITATIONS AND FUTURE WORK · p. 29
  • Healing requires a live process and accessible crash frame, as in Healer (Sun et al., 2026), while permitted HealCore calls may have side effects and failed submissions can leave state changes behind.Instrumentation must also preserve expected exception handling and keep evaluation code outside the healing scope.Source: E LIMITATIONS AND FUTURE WORK · p. 29
  • Uneven repository coverage, partial traces, and adapter-specific budget counting limit comparisons; target tests and path similarity do not establish semantic equivalence or verify that protected operations were prevented.Provisional simulation labels and manual-review case selection also limit how broadly decision accuracy can be generalized.Source: E LIMITATIONS AND FUTURE WORK · p. 29
  • Future work should improve source and sink coverage, reduce unnecessary rejection, contain state changes from failed healing, and validate instrumentation across more repositories and languages.The paper also calls for independently labeled cases and repeated matched executions to assess recovery correctness, prevented operations, component contributions, and runtime overhead, plus controlled comparisons isolating cross-file context.Source: E LIMITATIONS AND FUTURE WORK · p. 29

F.5 TOOL SCHEMAS AND EXECUTION FEEDBACK

  • The Codex response schema requires a reason field that the displayed prompt does not list.Source: §F.5 · p. 36
  • Execution feedback reports that code ran, not that healing was correct, and the tool distinguishes inspection with submit=false from committing healing with submit=true.A completion message does not verify healing correctness.Source: §F.5 · p. 36
  • Codex may extract embedded JSON after parsing fails, but complete schema revalidation is not established; historical inputs come from saved requests, and access-token requests are not reproduced.Source: §F.5 · p. 37
  • Submission errors return to the agent within its budget and timeouts, while OpenHands non-submission does not establish deliberate refusal.Source: §F.5 · p. 37

Related papers

Loading 2609.39086v1…