PaperlayerView exact revision on arXiv

Source-linked AI summary

Safety Must Survive Self-Improvement: Why Failures Persist and How Agents Recover

Yunbei Zhang, Janet Wang, Saiyue Lyu, Yingqiang Ge, Kaiqu Liang, Zijian Jin, Chandan K Reddy, Jihun Hamm

arXiv:2610.01073v1cs.SEcs.CR

TL;DR

Unsafe behavior can persist across generations even when a correct repair exists, making both deployment and recovery choices relevant to agent safety. The paper uses matched interventions in stateful authorization tasks to examine these choices. Historical scores retained unsafe programs in 22 of 48 histories, while full validation and validated rollback achieved fully correct endpoints with mean deployment service-cost savings above 43%, excluding validation expenditure.

  • Problem

    Agent evolution can preserve insecure reuse and behavioral drift, and a correct final program alone does not establish whether intervening execution remained safe.

    Source: §1 · p. 2

  • Method

    The study uses fixed LLM editors in a controlled testbed of stateful authorization tasks, with independent effect traces and matched interventions on evaluation, retention, and editing source.

    Source: §1 · p. 3, §2 · p. 4

  • Results

    Historical scores retained unsafe programs in 22 of 48 histories; full validation and validated rollback achieved fully correct endpoints with mean deployment service-cost savings above 43%, excluding validation expenditure.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Safety across generations depends on checking deployment candidates under current conditions and accounting for selection and retention as well as validation.

    Source: §5 · p. 12, §6 · p. 12

    The study uses controlled authorization tasks, fixed LLM editors, initially correct founders, externally specified contracts, and assumes validators are correct within tested conditions.

    Source: §6 · p. 12

Abstract

from arXiv · show

Recursive self-improvement (RSI) allows agents to carry useful changes across generations. Maintaining safety across these generations involves both preventing unsafe behavior from persisting and enabling recovery when failures occur. We study these challenges through a controlled testbed of stateful authorization tasks, where fixed LLM editors optimize executable agent components and independent traces record their effects. Paired interventions separate which revisions pass validation, which program continues running, and which program the editor revises next. After a new authorization dependency invalidates previously tested optimizations, historical scores preserve the same unsafe programs in 22 of 48 framework histories despite a correct alternative in every affected archive. Refreshing scores restores correctness on the original suite, with residual failures on independently composed tests. Failures also persist under an unchanged contract when all proposals are rejected and the failed incumbent remains active. Starting from shared failures, editing the initial implementation instead of the failed one improves recovery, although the advantage varies across editors. Full validation and validated rollback end with fully correct programs in the core trajectory study while retaining over 43% deployment savings. Preserving agent safety requires checking what will run under current conditions and choosing which implementation to edit next.

1 Introduction

The introduction frames safety across generations as a problem of both unsafe behavior persisting and recovery decisions. It presents a controlled testbed and matched interventions to separate validation, deployment, and editing choices.

1 Introduction

  • Recursive self-improvement makes safety across generations central because selected implementations can shape later requests and provide the starting point for subsequent revisions.Relevant changes can involve memory, tools, and workflows as well as model parameters.Source: §1 · pp. 1–2
  • A new authorization dependency can invalidate a previously tested optimization, while historical scores or keeping a rejected incumbent can leave unsafe execution active.The initially correct program checks effective permission on every request, unlike the optimization that relies on a local token version.Source: §1 · pp. 2–3
  • Matched comparisons separate historical from current evaluation, vary retention after rejection, and change the editing source while holding validation and retention fixed.The study identifies historical eligibility and keep-after-rejection as persistence mechanisms and examines editing source, validation, and fallback interventions.Source: §1 · p. 3
  • 22 of 48 framework histories retained the same unsafe programs despite correct archive alternatives; full validation and validated rollback reached fully correct endpoints with mean deployment service-cost savings above 43%, excluding validation expenditure.Refreshing scores restored correctness on the original suite, but independently composed tests exposed remaining coverage gaps.Source: §1 · p. 3
  • The testbed covers four stateful authorization families, uses reproducible event streams and independent effect traces, and rejects blanket refusal as successful repair.Fixed LLM editors repeatedly optimize executable components while histories evolve.Source: §1 · p. 3

2 Safety Across Generations

This section defines the evolving program, validation, retention, and recovery choices used to study safety across generations. It distinguishes current deployment eligibility from historical evaluation and measures safety over the full sequence.

2 Safety Across Generations

  • The study targets authorization safety of actions throughout repeated self-improvement, where fixed editors revise executable components and effects are recorded independently.The objective combines efficiency gains with full correctness, including authorized effects, expected outputs, and completion of permitted work.Source: §2 · p. 4
  • Each generation edits either the current incumbent or the initially correct founder, using an editing objective and available feedback to produce child programs.The incumbent is the current program; the founder is the initially correct program.Source: §2 · p. 4
  • A validation gate admits programs that pass the required tests, while retention rules determine which program is selected for subsequent execution.The selected program can differ from the program used as the editing parent.Source: §2 · p. 4
  • When no program passes, keeping the incumbent can leave a detected failure active; alternatives restore the founder, choose a historical program passing current checks, or pause execution.The historical-program option is validated rollback.Source: §2 · p. 4
  • Historical eligibility reuses recorded evaluations, whereas current validation checks eligibility under current tests and contract; an archive repair may remain unselected for editing or execution.A certificate records a suite pass, not a formal proof.Source: §2 · p. 4
  • Safety rate, safety restored, safe throughout, and full recovery track distinct outcomes, with full recovery requiring correctness after a violation and completion of permitted work.Unknown outcomes are not counted as safe.Source: §2 · p. 4

3 Experimental Evaluation

The experiments use matched interventions across stateful authorization tasks to isolate how validation, deployment choice, and editing source affect failure persistence and recovery. They show that safety controls can preserve useful optimization, but current test coverage and the choice of what runs and what gets revised remain consequential.

3.1 Setup

  • The testbed covers four stateful authorization families, with interleaved state changes and separate public, audit, and hidden tests.Fixed editors optimize under efficiency or maintenance objectives, and paired branches share starting programs and workloads.Source: §3.1 · p. 5

3.2 New Authorization Dependencies

  • Historical archive selection preserves unsafe programs after a new authorization dependency, while ordinary incumbent editing repairs unsafe checkpoints promptly.The dependency exposes unsafe behavior in 22 of 24 GPT checkpoints; frozen founders remain fully correct, and historical selection preserves the failure despite repair availability.Source: §3.2 · p. 5, Figure 2 · p. 6

3.3 Persistence through Historical Scores

  • Historical readout retains 22 unsafe programs through iteration 100 despite correct alternatives in every affected archive; current readout and score refresh restore correctness on the original suite.Independent event compositions expose residual failures, so refreshed eligibility and broader test coverage serve complementary purposes.Source: §3.3 · p. 7, Table 3 · p. 7

3.4 Persistence after Rejection

  • When the incumbent and both proposals fail, keep leaves the unsafe incumbent active, whereas founder fallback restores the correct founder.At full coverage, the remaining safety gap comes from pools without a passing program; the frozen-pool analysis finds 14 unsafe incumbents under keep.Source: §3.4 · p. 8, Figure 3 · p. 8

3.5 Editing Source and Recovery

  • After six generations from the same failed roots, current-code editing leaves 14 unsafe endpoints, compared with one under founder editing.The source advantage varies across editors, and the paired comparison does not establish that fuller feedback alone improves recovery.Source: §3.5 · p. 9, Figure 4 · p. 9

3.6 Safety and Utility

  • Validated rollback and full validation finish with fully correct endpoints while retaining mean service-cost savings above 43%.The study measures savings against the founder and excludes validation expenditure; an execution guard prevents unauthorized effects but does not ensure full correctness.Source: Figure 5 · p. 10, §3.6 · p. 11
  • In consent trajectories, founder editing yields higher final safety than inherited editing for each original editor, while approval shows a smaller effect and session and filesystem failures are rare.These trajectories begin with correct programs and allow each condition to generate its own failures; the shared-root study separately isolates recovery from identical failures.Source: §3.6 · pp. 10–11

4 Related Work

Prior work studies agent evolution, tool and memory risks, and safety in acting systems. This paper focuses on how evaluation, retention, and editing-source decisions affect behavior that persists across generations.

Recursive self-improvement

Agent safety

5 Discussion

The discussion distinguishes current deployment eligibility from historical test results and separates what executes from what the editor revises. It also notes that safety and utility assessments remain bounded by test coverage and validation cost.

5 Discussion

  • A new dependency can change which actions are permitted, so historical passes record tested conditions rather than current authorization.The archive may contain a correct repair even while stored rankings favor an invalid program.Source: §5 · p. 12
  • Deployment fallback and editing source are separate choices: fallback protects the next action, while editing source determines where the next repair attempt starts.Protection is still needed during repair, including when stronger editors shorten some repair processes.Source: §5 · p. 12
  • Safety evaluation is bounded by test coverage because new event combinations can expose failures in programs that pass the current suite.The discussion recommends recording selected programs, authorization of their actions, tested conditions, and final task scores across generations.Source: §5 · p. 12

6 Conclusion

The paper treats safety across recursive self-improvement as a problem of detecting failures, making repairs available, and ending unsafe execution. Its controlled authorization testbed distinguishes those stages while examining safety, correctness, and deployment cost.

6 Conclusion

  • Safety depends on selection and retention as well as validation: a tested optimization can become unsafe after a new authorization dependency, and a rejected failure can remain active.Current deployment validation, refreshed archive scores, and validated fallback improve safety while preserving efficiency gains in the studied tasks.Source: §6 · p. 12
  • The study is limited to controlled authorization tasks, fixed LLM editors, initially correct founders, and externally specified contracts, and assumes validators are correct within tested conditions.Larger deployments, longer histories, and jointly evolving editors remain future directions.Source: §6 · p. 12

A.1 Dataset Construction

  • Each instance is an ordered event stream, with setup and updates changing the environment and requests asking the program to perform protected actions.The core splits use 24 public, 24 audit, and 24 hidden streams per block, covering eight categories with three probes each.Source: §A.1 · p. 18, Table 4 · p. 19
  • The testbed pairs stateful authorization tasks with seeded event streams and independent effect traces, distinguishing unauthorized actions from failure to complete permitted work.It covers session authorization, filesystem containment, tool approval, and structured user consent.Source: §A.1 · pp. 18–19

A.2 Contracts and Execution

  • Full correctness requires expected outputs and authorized effects, including completing permitted requests; blanket refusal cannot count as successful repair.Independent records connect requests to effects, costs, and program identity.Source: §A.1 · p. 18, §A.2 · p. 20

A.3 Cost Accounting

  • Deployment savings compare final routine service cost with founder cost, while validation is accounted separately from deployment and model-generation costs.These are task-service units, not wall-clock latency or API expenditure.Source: §A.3 · p. 21

A.4 Statistical Inference

The paper uses paired comparisons to separate changes in editing source, feedback, and retention from variation in roots and evaluation. In the consent task, founder editing improves endpoint safety over inherited efficiency, while the recovery study also favors editing from the founder.

A.4 Statistical Inference

  • Inference pairs blocks by model, query cost, and seed, resamples within model strata, and adjusts the declared family of 16 contrasts with Holm correction.The paired tests condition on discordant endpoint outcomes; generations and proposed programs are not treated as independent samples.Source: §A.4 · p. 21, Figure 6 · p. 22
  • Founder editing improves consent endpoint safety over inherited efficiency by 56.9 percentage points (95% CI 45.8 to 68.1, Holm-adjusted p<0.001).Ever-unsafe counts are similar: 63 and 61 among 72 lineages per condition.Source: §A.4 · p. 21
  • Editing from the founder instead of the failed implementation reduces unsafe endpoints by 31.0 percentage points (95% CI 23.8 to 38.1; Holm-adjusted p=0.000488).The contrast has 13 improved roots and no worsening.Source: §A.4 · p. 21

B.1 Conditions and Policies

  • The changed-authorization study crosses three editors, two mechanisms, and 12 seeds; its eight strategies share four initial generations before varying editing source, retention, or archive eligibility.The cohort expanded by eight seeds after pilot inspection; GPT had unsafe frozen checkpoints in seven of eight pilot cases and 15 of 16 expansion cases.Source: §B.1 · pp. 22–23

B.2 Eligibility and Coverage

Deployment revalidation changes which program runs without changing the matched parents or proposals, improving safety after authorization conditions change. The section also shows that refreshed or older certificates do not eliminate coverage gaps or replace current evaluation.

B.2 Eligibility and Coverage

  • At the first post-change selection, deployment revalidation improves safety by 30.6 percentage points (95% CI 26.4 to 33.3) and full correctness by 55.6 points (CI 48.6 to 62.5).The paired policies use identical parents and proposals in every block.Source: §B.2 · p. 23
  • Independent event compositions reveal failures that remain after scores are refreshed, and a passing new proposal is already available in 123 of 132 unsafe historical selections.Current deployment validation retains 36.4% mean founder-relative service savings while mean validation probes rise from 439.0 to 540.5 per history.Source: §B.2 · pp. 23–24

B.3 Old Certificates

  • Replaying old-certified selections under the changed contract yields 20 unsafe programs, 37 fully correct programs, and 15 other incorrect programs.In the sole all-candidates-rejected pool among 432, current revalidation selects a correct historical version while keep and old-certificate fallback choose one that misses permitted work.Source: §B.3 · p. 24, Table 7 · p. 24

C.1 Execution Scope

  • The framework cohorts compare historical and current readouts of retained programs, while score refresh updates public scores, feedback, and score-dependent selection metadata before evolution resumes.Independent evaluation uses frozen streams with deeper dependencies, multiple tokens, and resource interleavings; these outcomes do not enter prompts, ranking, or feedback.Source: §C.1 · pp. 25–26

C.2 Independent Evaluation

Independent compositions show that score refresh reduces unsafe selections but does not remove all coverage errors. Other experiments distinguish recovery by stopping or restoring a passing program from keeping a failed incumbent active, and assess safety alongside task completion and service savings.

C.2 Independent Evaluation

  • Across 48 long histories, current readout is unsafe at least once in eight, becomes unsafe after a correct selection in six, and recovers in three.The long histories and 48 refresh branches span different iteration counts, so their recurrence counts are not compared over equal observation periods.Source: §C.2 · p. 26
  • On independent compositions, score refresh reduces unsafe selections from 25 to three and raises fully correct endpoints from six to 43 at matched iteration-60 endpoints.The paired safety contrast has 23 improvements and one worsening; the table reports endpoints by framework, editor, and task.Source: §C.2 · p. 26, Table 9 · p. 27

C.3 No Valid History

  • When every historical program is invalid, 23 of 24 DGM-derived histories recover after one generation and the last after two; keep executes one unsafe selected generation, while stop pauses it.Pausing prevents unauthorized execution during repair but does not count as correct task completion.Source: §C.3 · p. 27

C.4 Correctness and Efficiency

  • Among the 43 refresh endpoints correct on independent streams, median service savings are 16.9% against one authoritative check per request.Across all 48 endpoints, the corresponding median is 16.6%, or 47.9% against the redundant founder; correct endpoints are reported separately to distinguish efficient completion from omitted permitted work.Source: §C.4 · p. 28

D.1 Fixed Candidate Pools

  • In fixed candidate pools, 25 have no passing program at full coverage; founder fallback selects a correct program in every such pool.Fourteen pools retain unsafe incumbents and 11 retain other incorrect programs under keep.Source: §D.1 · p. 28

D.2 Retained Failures and Complete Histories

  • From generation 4, full validation rejects every subsequently selected unsafe program, but keep still runs the unchanged incumbent when all candidates fail.Across 198 histories, all 346 unsafe states have rejected incumbents and proposals; GPT remains correct in all 792 selected states, while Qwen ends unsafe in 33 of 66 branches and Sonnet finishes correctly in all 66.Source: §D.2 · pp. 28–29, Figure 8 · p. 29

D.3 Shared Failed Roots

  • The recovery study pairs editing-source comparisons on shared roots selected after prior failures, with hidden probes reserved for outcome analysis.The original editors cross editing source, feedback, and retention; GPT and Sonnet each process all 42 roots under three conditions.Source: §D.3 · p. 29

D.4 Prevention

Full validation prevents unsafe execution in the Sonnet prevention comparison, while validated rollback retains efficiency gains with fully correct programs. Across later trajectories, safety and service savings vary by editing strategy, and cost reductions often accompany entry into unsafe execution.

D.4 Prevention

  • 14 of 24 Sonnet scheduled-current lineages were ever unsafe versus zero under full validation.The scheduled-current arm also had 32 unsafe states and one other incorrect state; full correctness requires permitted work as well as avoiding unauthorized effects.Source: §D.4 · pp. 30–31

E.1 Efficiency Across Generations

  • 28.1% consent savings at the validated-rollback endpoint followed 9.9% after the first edit, with every selected generation fully correct.Endpoint savings were 21.7% for approval, 78.0% for session, and 46.2% for filesystem tasks.Source: §E.1 · p. 31, Figure 9 · p. 32
  • 70.6% consent savings at generation 3 coincided with 15.3% safety under inherited efficiency; by generation 12, savings were 62.2% and safety 33.3%.Founder editing ended at 34.0% savings and 90.3% safety.Source: §E.1 · p. 31

E.2 Editor Differences

  • 30 maintenance-inheritance consent endpoints were unsafe versus 48 under efficiency inheritance, a paired difference of 25.0 points (95% CI 15.3 to 36.1).For consent, founder editing ended with fewer unsafe programs than inherited efficiency for all three original editors.Source: §E.2 · pp. 31–32, Figure 10 · p. 33

E.3 Costs and Ties

  • At 100 workload batches, validated rollback had a mean cost ratio of 0.648 and full validation 0.628; at 10,000 batches, ratios were 0.618 and 0.604.Ratios below one indicate savings against the founder baseline; the workload volumes are sensitivity scenarios.Source: §E.3 · p. 32, Figure 11 · p. 33
  • 60 of 61 entries into unsafe consent execution coincided with strict cost reductions, while 564 of 626 unsafe states had unchanged costs.These are descriptive state counts, not independent samples.Source: Figure 12 · p. 34, §E.3 · p. 34

E.4 Factor Controls

Factor controls separate proposal generation, cost-based selection, and fallback choices, while recorded histories show how failures persist after rejection. Outcomes differ across editors, and paired endpoint tests do not establish safety differences for the prompt/ranking, price, or tie comparisons.

E.4 Factor Controls

  • The 36 starting blocks generated 198 condition branches and 2,376 post-edit states, with audit and hidden outcomes excluded from feedback and selection.The gate used 12 routine streams in generations 1–3 and all 24 thereafter.Source: §E.4 · pp. 35–36
  • Under GPT efficiency instructions, consent savings rose from 40.3% to 75.8% with cost ranking while every selection stayed correct.Sonnet repaired its intermediate failures, while persistent Qwen failures also arose under maintenance instructions.Source: §E.4 · p. 35
  • 83 of 88 transitions into unsafe execution coincided with strict cost reductions and five with ties; incumbent preference left paired endpoint safety unchanged.Consent unsafe states increased from 64 to 66 and approval from four to five under incumbent tie preference.Source: §E.4 · p. 35, Table 17 · p. 37
  • 479 unsafe selected states occurred for Qwen, 77 for Sonnet, and zero for GPT; 345, one, and zero, respectively, occurred after full validation began.All 346 post-gate unsafe states retained the preceding program’s code identity after rejection.Source: §E.4 · p. 36, Table 18 · p. 37

F Recorded Examples

  • Recorded cases pair retained source code with behavior to explain aggregate comparisons, including stale historical eligibility, residual coverage failure, and consent repair.They are explanatory cases, not independent estimates of failure frequency.Source: F Recorded Examples · p. 37

F.1 Delegation Cache

Recorded examples trace authorization failures to stale cached decisions and show repairs that recheck changing state. The paper also examines native workflow transfer and positions its controlled selection experiments alongside work on recursive self-improvement, reuse, and safety during adaptation.

F.1 Delegation Cache

  • A delegation program kept using a token-local version after upstream revocation changed effective permission, despite correct alternatives already in the archive.Its historical score was 0.10765 against a current score of zero; refreshing selection immediately chose a valid later program.Source: §F.1 · pp. 37–38

F.2 Casbin Policy Cache

  • A Casbin policy change invalidated permission without refreshing cached state, so historical selection retained an unsafe program despite a later repair.Current selection chose a valid later program.Source: §F.2 · p. 38

F.3 Unseen Interleaving

  • A shared version update for one request left another request’s stored decision stale, causing unauthorized execution on independent interleavings.The endpoint passed the original suite but failed independent interleavings.Source: §F.3 · p. 38

F.4 Consent Repair

  • Checking saved consent against the current epoch restored the four correct outcomes, while user checks rose from one to four and service cost from 16 to 50.The repaired program passed every public and hidden probe and remained correct for all six generations.Source: §F.4 · pp. 39–40, Figure 13 · p. 39

G Native Dialogue Transfer

  • In native dialogue transfer, unauthorized-write proposals occurred after delayed withdrawal, but the paired comparison did not establish that evolution made them more frequent.The runs issued no writes; inherited efficiency changed from 16 of 168 proposals to 14 of 168 by generation 6.Source: G Native Dialogue Transfer · p. 40

H.1 Recursive Self-Improvement

  • The paper studies repeated editing of executable agent components with fixed editors, separating its focus from systems that change model parameters or improvement procedures.Its source audit distinguishes execution of upstream components from inspection of other systems.Source: §H.1 · p. 41

H.2 Feedback, Memory, and Skills

  • Reusable skills and memories make future decisions depend on retained content; the permission-cache cases test whether past authorization dependencies still hold when reused.The controlled experiments do not require poisoned memory or adversarial editing instructions for invalid reuse to arise.Source: §H.2 · p. 42, §H.3 · p. 42

H.3 Safety During Adaptation; H.4 Objectives and Parameter Adaptation

  • Matched interventions add evidence on selection and editing context to related work on evolving-system risks and evaluation mismatches.The experiments compare archives under different eligibility rules, rejected candidates under different fallbacks, and failed roots under different editing sources.Source: §H.3 · p. 42, §H.4 · p. 42

H.5 Evaluation and Enforcement

The paper distinguishes authorized effects from functional task completion and compares its controlled authorization contracts with broader agent-risk benchmarks. Its execution guard blocks unauthorized effects but does not repair evolved code that omits permitted work.

H.5 Evaluation and Enforcement

  • Full correctness requires legitimate completion as well as authorized effects, distinguishing failures that risk-only evaluation can miss.Following selected programs across generations separates never entering failure, retaining it, repairing it, and failing again after recovery.Source: §H.5 · p. 43
  • Broader benchmarks cover risks and environments beyond the paper’s controlled contracts, while native workflow transfer outcomes support keeping task success and authorization behavior separate.The native transfer uses released workflows that evaluate interactive task completion under domain policies.Source: §H.5 · p. 43

H.5 Evaluation and Enforcement; I Scope and Limitations

  • The execution guard blocks unauthorized effects but does not repair code that omits permitted work; recovery also depends on which code is edited next.The core controls assume a correct founder, externally specified contract, and correct validator; the validator implementation is not varied.Source: §H.5 · p. 43, I Scope and Limitations · p. 44

Related papers

Loading 2610.01073v1…