PaperlayerView exact revision on arXiv

Source-linked AI summary

FIRE: Failure-Informed Runtime Engineering for Reliable Language-Model Agents

Nikita Agarwal, Nivedit Jain

arXiv:2609.26048v1cs.AIcs.CLcs.SE

TL;DR

Agents can reach a solution yet lose it during delivery, and comparisons against no intervention do not distinguish targeted instructions from the effect of interruption. FIRE derives targeted runtime policies from observed failure states and evaluates them against a timing-matched sham, generic checks, and silent tasks in a randomized five-arm panel. On 14 eligible tasks, real policies passed 17/28 attempts (60.7%) versus 10/28 (35.7%) for triggered sham; for Sol, pass∧2 rose by 9.2 points.

  • Problem

    Agents can reach a solution yet lose it during delivery, and comparisons against no intervention do not distinguish targeted instructions from the effect of interruption.

    Source: §1 · p. 1

  • Method

    FIRE derives targeted runtime policies from observed failure states and evaluates them against a timing-matched sham, generic checks, and silent tasks in a randomized five-arm panel.

    Source: §1 · pp. 1–2

  • Results

    On 14 eligible tasks, real policies passed 17/28 attempts (60.7%) versus 10/28 (35.7%) for triggered sham; for Sol, pass∧2 rose by 9.2 points.

    Source: §5.1 · p. 5, §5.4 · p. 6

  • Takeaways & Limitations

    Runtime policies can improve repeatability without retraining, but they target procedural losses rather than supplying missing algorithms or domain knowledge.

    Source: §1 · p. 2

    Evidence is limited to one English-language benchmark, one harness, one model family, and one reasoning effort; two attempts do not estimate long-run task reliability.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Language-model agents often reach a working solution and then fail to consistently deliver it. We study runtime policies: targeted natural-language instructions and action denials applied by the agent harness at states that preceded observed failures, without changing model weights or the user prompt. With this, keeping capability constant, we observe a meaningful unlock in delivered reliability. Across the complete 87-task Terminal-Bench 2.1 suite, with two attempts per task, policies increase repeated success (pass^2) in all three GPT-5.6 tiers: 50.6% to 54.0% for Luna, 55.2% to 60.9% for Terra, and 64.4% to 73.6% for Sol. Sol's best-of-two success changes by 1.2 points while repeated success rises by 9.2, showing that policies chiefly convert reachable solutions into dependable delivery. We further cover 14 tasks under Terra's frozen portfolio. Policy-guided Terra reaches 71.4%, compared with 64.3% for unassisted Sol, at about half the cost, demonstrating how engineering around models could unlock dependability for a use case. To isolate the mechanism we run a randomized five-arm experiment: real policies reach 61% on eligible tasks, versus 39% without a policy, 36% with a timing-matched sham, and 39 to 43% with generic verification or reconsideration. The intended corrective behavior appears in 22 of 24 coded policy attempts, against at most 14 in any other arm. Runtime policies are therefore a practical reliability layer: they make capabilities an agent already possesses substantially more repeatable.

1 Introduction

Agents can produce working solutions yet lose them during delivery, and standard intervention comparisons cannot distinguish targeted policy content from the effect of interruption. FIRE derives policies from observed failures and tests their content against timed and generic controls.

1 Introduction

  • 16–19 tasks per model tier pass in exactly one of two policy-free attempts, illustrating how delivery failures can erase an otherwise reachable solution.Examples include a server that exits with the agent’s shell, a database altered before its log is copied, and protected bytes changed by formatting.Source: §1 · p. 1
  • Standard comparisons with no intervention cannot separate the effect of a targeted rule from simply interrupting an agent to review its work.Prior approaches include self-checking and runtime enforcement (Shinn et al., 2023; Snell et al., 2025; Rebedea et al., 2023; Wang et al., 2026; Shi et al., 2025).Source: §1 · p. 1
  • FIRE derives policies from observed failures and tests them with a randomized five-arm panel that includes a timing-matched sham, generic checks, and silent tasks.The attempt-level funnel records the risky state, policy firing, registered behavior, and verifier outcome.Source: §1 · pp. 1–2
  • The paper frames reliability as a systems property: targeted runtime policies can improve repeatability without retraining, while not supplying missing capability.Its contributions include randomized attribution, a mechanism funnel, replication, complete-suite results, and a comparison of policy-guided Terra with unassisted Sol on covered tasks.Source: §1 · p. 2

2 Related Work

Related work spans interactive-agent evaluation, self-correction and test-time computation, and runtime enforcement. FIRE distinguishes its approach by deriving rules from observed task failures and testing whether policy text—not merely triggering—changes delivery behavior.

2 Related Work

  • Interactive-agent benchmarks cover varied settings, while pass∧k measures success on all k trials and cost, scaffold, and run-to-run variation are evaluation concerns.The paper reports pass∧2 alongside pass@k.Source: §2 · p. 2
  • Self-correction methods revise behavior using feedback, but whether generic requests to reconsider help remains contested.The paper’s always-verify and reconsider arms are minimal members of this family, tested under the same harness.Source: §2 · p. 2
  • Runtime-enforcement work includes trigger-based rules, privilege restrictions, and model-based checking, often focused on preventing unsafe actions.FIRE derives rules from observed task failures, targets delivery, and tests whether rule text rather than triggering accounts for improvement.Source: §2 · p. 2
  • In the randomized panel, only the real portfolio raises success on eligible tasks; the triggered sham and generic controls do not help.The panel also includes 16 silent tasks where the policy never fires and success is unchanged.Source: Figure 1 · p. 3

3 Runtime Policies Derived from Failures

The paper targets procedural losses: cases where an agent demonstrates task capability but an observable, preventable execution step causes failure. It derives bounded runtime policies from those failure states and freezes a screened portfolio.

3 Runtime Policies Derived from Failures

  • A procedural loss requires evidence of capability, an observable pre-failure state, and a bounded harness intervention that does not supply the solution.Policies target procedural losses rather than missing algorithms or domain knowledge.Source: §3 · p. 3
  • A runtime policy separates task eligibility from the runtime trigger, intervention, release condition, and tracking state.Eligibility reads the task description; runtime predicates inspect action history and policy state at harness events, while interventions instruct or deny actions.Source: §3 · p. 3
  • For service persistence, the policy checks for detached launch and a later functional probe, then instructs relaunching and probing if the agent tries to stop without both.The policy enforces a lifecycle invariant without writing the server or choosing its protocol; Figure 2 shows the server dying with the baseline session and surviving the policy-guided relaunch.Source: §3 · p. 3, Figure 2 · p. 4
  • Policies are derived by inspecting failed policy-free attempts, grouping concrete procedural omissions by mechanism, and screening candidate rules on positive and nearby-negative tasks.Candidates are revised or retired if inert, over-broad, or harmful; portfolio families require an observable failure state, a focused positive result, and acceptable routing.Source: §3 · pp. 3–4
  • The frozen Terra portfolio contains eight families in seven source files and 11 rules, while Luna shares five sources and Sol uses five broader stop-time families.No source contains task names, fixture paths, or expected answers.Source: §3 · p. 4

4 Evaluation Design

The evaluation uses fixed model routes and a randomized five-arm panel to distinguish policy content from interruption and generic checking. Its inference treats tasks—not attempts—as the unit of analysis.

4 Evaluation Design

  • The evaluation uses English-language Terminal-Bench 2.1 tasks, Harbor containers, Codex CLI at medium reasoning effort, and three GPT-5.6 routes called Luna, Terra, and Sol.Timeouts and agent failures count as failures; reasoning effort, timeouts, and nudge limits are fixed across arms.Source: §4 · p. 4
  • The five-arm panel tests 14 Terra-eligible and 16 silent tasks, with two attempts per task and condition, randomized condition order, and 300 attempts total.The triggered sham matches policy eligibility and timing with generic review text; always-verify and reconsider provide generic-checking controls.Source: §4 · p. 4
  • Table 1 presents each frozen Terra policy family as a registered chain linking a watched runtime state, intervention, and pre-registered behavior marker.The table’s predicates, hashes, and source are in Appendices G–H.Source: Table 1 · p. 5
  • The panel’s primary contrast is real policy versus triggered sham on eligible tasks, with 95% task-level bootstrap intervals and paired sign-flip tests.The 14-task panel yields discrete permutation p-values, so effect sizes, intervals, and tests are reported together.Source: §4 · p. 5

5 Results

Across the complete suite, policies chiefly improve repeated delivery, while a randomized panel and mechanism analysis distinguish targeted policy text from interruption or generic checking. On Terra’s 14 covered tasks, policy-guided Terra exceeds the unassisted Sol baseline at lower cost, but the estimate is imprecise.

5.1 Policy text, not timing or generic checking, drives the gain

  • On 14 eligible tasks, the real portfolio passes 17/28 attempts (60.7%), versus 10/28 (35.7%) for the triggered sham; real minus baseline is +21.4 points (p = 0.032).The primary real-minus-sham contrast is +25.0 points (95% CI [7.1, 46.4], p = 0.061), a large but imprecisely estimated effect; generic verification and reconsideration end near baseline.Source: §5.1 · p. 5, Table 3 · p. 6

5.2 Same opportunity, different behavior

  • The risky runtime state arises in 24–26 of 28 attempts in every arm, but registered corrective behavior appears in 22 of 24 coded real-policy attempts versus 11–14 of 24 elsewhere.The behavior codes use deterministic extractors applied identically across arms; four attempts per arm are excluded from that column.Source: §5.2 · p. 6, Table 4 · p. 6

5.3 One policy replicates in two further tiers

  • On a registered seven-task screen, service-persistence success rises from 7/21 to 19/21 for Sol and from 11/21 to 19/21 for Luna, with no pass-to-fail changes.The authors note that seven tasks allow little inference; exact sign-flip p-values are 0.063 for Sol and 0.125 for Luna.Source: §5.3 · p. 6

5.4 Complete suite: consistent direction, gains in repeated success

  • For Sol, pass∧2 rises by 9.2 points while pass@2 rises by only 1.2 points; Terra improves reach (+6.9 points) and repeated success (+5.7 points), and Luna rises on all three metrics.Across the complete 87-task suite, policies improve pass@1 in all three tiers; the portfolios differ by tier and are not pooled.Source: §5.4 · pp. 6–7, Table 5 · p. 7

5.5 Policies bring Terra to the Sol baseline on covered tasks

  • On Terra’s 14 covered tasks, policies raise success to 20/28 (71.4%) from 10/28 (35.7%), versus 18/28 (64.3%) for unassisted Sol, at $7.19 rather than $14.48.The estimate is imprecise (95% task-bootstrap CI [−21.4, 35.7], p = 0.82), so it does not establish that Terra is generally superior to Sol.Source: §5.5 · p. 7

6 Discussion

Runtime policies improve repeatable delivery across the benchmark and can bring policy-guided Terra into Sol’s performance range on covered tasks at lower cost.

6 Discussion

  • Sol’s pass∧2 rises by 9.2 points while pass@2 remains nearly flat, indicating more repeatable delivery across the complete suite.The randomized panel shows the same pattern at the attempt level.Source: §6 · p. 7
  • Across the complete suite, policy-guided Terra reaches 70.7%, closing much of the gap to the 75.3% Sol baseline.Source: §6 · p. 7
  • On the 14 tasks covered by Terra’s portfolio, policy-guided Terra reaches 71.4% versus 64.3% for unassisted Sol, at approximately half the cost.Source: §6 · p. 7

7 Conclusion

Runtime policies address procedural failures by making capable agents’ solutions more repeatable. The results show improved repeated success and a covered-task comparison with unassisted Sol at about half the cost.

7 Conclusion

  • Across two attempts on the full 87*-task suite, policies increase repeated success in all three model tiers.Source: §7 · p. 7
  • Sol’s pass∧2 rises from 64.4% to 73.6% while pass@2 is nearly flat.Source: §7 · p. 7
  • On tasks covered by Terra’s portfolio, policy-guided Terra reaches the same performance range as unassisted Sol at about half the cost.Source: §7 · p. 7
  • Randomized controls indicate that policy content changes decisive behavior, unlike an identically timed interruption or generic requests to verify or reconsider.Source: §7 · p. 7

Limitations

The evidence has bounded scope: two-attempt observations on one benchmark, harness, model family, and reasoning effort do not establish long-run reliability or broad generalization.

Limitations

  • Two attempts per task make pass∧2 an observed rate, not an estimate of each task’s long-run reliability.Source: Limitations · p. 8
  • Evidence comes from one English-language benchmark, the Codex CLI harness, one model family, and one reasoning effort.Source: Limitations · p. 8
  • The frozen randomized panel supports attribution on its development-population distribution, but does not establish generalization.Source: Limitations · p. 8
  • Tier comparisons use model-specific portfolios, and the planned leave-one-family-out ablation was not run.Source: Limitations · p. 8
  • Policies can add latency, disturb prompt caching, and, when broad, cancel their own gains.Source: Limitations · p. 8

Ethical Considerations

The evaluation used isolated containers without human subjects or personal data, while the authors flag risks from runtime action denials and more durable agent actions.

Ethical Considerations

  • The evaluated policies deny destructive operations on task evidence or deployed state, but runtime denials could also restrict actions users intend.The authors say deployed policies should be inspectable by the people the agent acts for.Source: Ethical Considerations · p. 8
  • Policies that keep services running make actions more durable, raising the stakes when an agent acts on a mistaken goal.Source: Ethical Considerations · p. 8
  • The work used isolated containers and involved no human subjects or personal data.Source: Ethical Considerations · p. 8
  • Policy source, configuration, run-selection records, per-attempt outcomes, and analysis outputs are released for inspection.Source: Ethical Considerations · p. 8
  • Both authors are Failproof AI cofounders and have financial and professional interests in the company.Source: Ethical Considerations · p. 8

A Reproducibility

The section documents the released artifact, the task-level statistical procedure, and recovery rules for control-panel observations. Complete-suite transitions show how often Sol’s observed outcomes changed under policies.

A Reproducibility

  • The artifact includes configuration and run-selection records, per-attempt outcomes and behavior labels, frozen policy sources, lineage records, and machine-readable analyses.The records support recomputation of aggregate results without querying the model again.Source: A Reproducibility · p. 10

B Statistical Procedure

  • Inference averages two attempts per task and condition, then uses paired task-level differences, bootstrap intervals, and sign-flip permutation tests.Attempts within one task are not treated as independent; the interval and permutation test can disagree in small discrete samples.Source: B Statistical Procedure · p. 10

C Control Panel Lineage and Deviations

  • For replicate 2, a frozen recovery rule keeps each original fair observation or selects the first fair result in chronological recovery order.The lineage audit records candidate results and decisions; repeated provider failure on a task counts as failure rather than triggering selective retries.Source: C Control Panel Lineage and Deviations · p. 10

D Complete-Suite Details

  • Of Sol’s 19 baseline 1/2 tasks, six become 2/2 and three become 0/2, while 55 of 56 baseline 2/2 tasks stay there.Table 6 organizes transitions by baseline successes in rows and policy successes in columns.Source: D Complete-Suite Details · p. 10, Table 6 · p. 10

E Service-Persistence Screens

The service-persistence screens report policy outcomes and show task-level movement, while the authoring account describes how policies were narrowed, retired, and bounded by observable intervention points.

E Service-Persistence Screens

  • On the 14 tasks eligible for Terra’s portfolio, Terra with policies was 7.1 points above policy-free Sol, with a 95% CI of [−21.4, 35.7] and p = 0.82.The caption identifies two attempts per task and notes that each tier used its own portfolio.Source: Table 8 · p. 11
  • Sol’s give-ups fell from 14 to 2, while its remaining failures were deploy-hook logic failures with the service reachable, outside the policy’s mechanism.The matched service-persistence screen used three attempts per task and a same-day policy-free baseline.Source: E Service-Persistence Screens · p. 11, Table 9 · p. 11

F Policy Authoring and Retirement

  • Policy authoring retired or revised prompts after inert, adverse, or misrouted results, and left some tasks uncovered when no reliable harness-observable intervention point was found.Large-scale editing, video processing, and query optimization were classified as capability gaps; researcher hours were not logged contemporaneously.Source: F Policy Authoring and Retirement · p. 11

G Frozen Terra Policy Specification

This section specifies the frozen Terra runtime-policy setup and its source, covering targeted, task-specific checks for service deployments, database evidence, exact replacements, LaTeX logs, binary equivalence, and dataset-token accounting. The policies emphasize mechanically checkable constraints, including reconciling token counts and enumerating valid answers before stopping.

G Frozen Terra Policy Specification

  • The Terra profile tracks eligible rules at prompt, tool, and stop events, logs co-firing, and repeats the database-evidence denial until a sufficient copy exists.Policy attempts are not scored if hook events, policy loading, or required model and profile markers are missing.Source: G Frozen Terra Policy Specification · p. 11

H Complete Terra Runtime-Policy Source

H.1 Service persistence and deployment preservation

  • The service rules require a persistent-service contract and recognizable launch command, then prompt a detached launch and a separate later probe before finishing.After a successful evaluator-facing deployment probe, a separate rule blocks cleanup that would alter the tested deployment state.Source: §H.1 · pp. 12–13

H.2 Database evidence preservation

  • The database-evidence rule requires a byte-preserving copy of the artifact and sidecars before mutating or opening it, because copying only main.db can lose useful WAL evidence.The denial repeats until a sufficient backup exists.Source: §H.2 · p. 13

H.3 Exact replacements

  • The exact-replacements subsection identifies a narrow credential path as the focus of the frozen policy.Source: §H.3 · p. 13
  • Frozen exact-replacement policies constrain edits to explicitly mapped tokens and require mechanical diff checks that preserve surrounding bytes and EOF-newline state.They also direct agents to verify placeholders and original-value absence, limit changes to contaminated files, and avoid unrelated edits.Source: §H.3 · pp. 14–15

H.4 LaTeX log validation

  • The LaTeX-log policy requires checking the final compile log for the literal “Overfull \\hbox,” rather than inferring cleanliness from exit status.If present, only an explicitly authorized replacement is allowed, followed by another compile and literal log check.Source: §H.4 · p. 15

H.5 Binary equivalence

  • The binary-equivalence policy requires immediate candidate-versus-original testing with the evaluator’s build and invocation, comparing output artifacts and streams byte-for-byte.Before stopping, it calls for validation over diverse inputs and a method change if equivalence or an acceptance bar is missed.Source: §H.5 · p. 16

H.6 Dataset-token accounting

  • The dataset-token-accounting policy is described as a narrow successor to a broad requirement ledger that preserves the field-reconciliation intervention that recovered the token-counting target.Source: §H.6 · p. 16
  • For dataset token counts, inspect the actual selection, count each contributing text field with the named tokenizer, and reconcile field subtotals to the final integer.Use an explicitly documented mapping only when the exact requested value is absent, verify the selection is non-empty, and do not report one field as the total.Source: §H.6 · p. 16

H.7 Complete multi-answer enumeration

  • When the output contract requires every valid answer, enumerate and test the complete candidate set, remove duplicates, and emit all survivors in the requested format.Before stopping, verify that no additional candidate satisfies the same winning or validity condition.Source: §H.7 · p. 17

Related papers

Loading 2609.26048v1…