PaperlayerView exact revision on arXiv

Source-linked AI summary

Evolutionary Safety of Recursive Self-Improving AI: Taxonomy, Risk Discovery, and Evaluation

Chang Gong, Jingping Bi, Di Yao, Xinjian Liang, Chao Xiang, Ruijie Guo

arXiv:2609.31186v1cs.AI

TL;DR

Existing accounts do not yet unify how safety-relevant changes are generated, selected, retained, and inherited. The paper develops a framework for reasoning about safety under persistent and recursive self-improvement, concluding that safety becomes evolutionary when changes are retained, selected, and inherited.

  • Problem

    Existing accounts do not yet unify how safety-relevant changes are generated, selected, retained, and inherited.

    Source: §1 · p. 2

  • Method

    The paper frames safety under persistent and recursive self-improvement and organizes evolutionary risks by their carriers and temporal pathways.

    Source: §1 · p. 3

  • Results

    Safety becomes evolutionary when changes are retained, selected, and inherited.

    Source: §8 · p. 19

  • Takeaways & Limitations

    Lineage traceability links each accepted change to its source, test evidence, verifier, and permission.

    Source: §6 · p. 18

    An accepted candidate can reduce capability or weaken safety.

    Source: §2.1 · p. 5

Abstract

from arXiv · show

Artificial intelligence is advancing rapidly, with increasingly capable systems taking larger roles in reasoning, decision-making, scientific discovery, and autonomous development. As AI begins to participate in its own improvement, from model training and experience accumulation to agent evolution and automated AI development, the prospect of recursive self-improvement (RSI) is becoming increasingly relevant. This transition raises a fundamental safety question: how can safety be maintained when the system, its accumulated experience, and even the process producing its successors continue to change? We introduce Evolutionary Safety as a perspective for studying safety under persistent and recursive self-improvement. It concerns not only whether an AI system is safe at a particular moment, but how safety properties change, persist, accumulate, and propagate throughout evolution. We characterize recurring manifestations, including intent drift, error accumulation, experience contamination, safety-property erosion, evaluator drift, and risk propagation. We then develop a taxonomy spanning persistent agent state, model state, evaluation and environmental feedback, computational substrate, and meta-level update mechanisms. Building on this taxonomy, we examine how evolutionary risks can be discovered and evaluated across states, updates, trajectories, and lineages, and derive governance principles for modification, selection, authorization, provenance, and recovery. Finally, we outline open problems toward maintaining safety guarantees as AI systems become increasingly persistent, adaptive, and recursively self-improving. Project resources and proposed evaluation systems are available at https://chaunceykung.github.io/evolutionary-safety-rsi.

1 Introduction

The introduction frames safety as a property that can change and propagate as AI systems retain experience, update themselves, and produce successors. It proposes Evolutionary Safety to connect those changes to their carriers, evaluation, and governance.

1 Introduction

  • The perspective identifies a gap: existing work does not yet offer a unified account of how safety-relevant changes are generated, selected, retained, inherited, and potentially amplified across successive rounds of self-improvement.Examples include feedback carried into later attempts, executable skill libraries, reusable procedures, tool revision, and evaluator participation in the improvement loop [15] [106] [113] [119] [120] [127].Source: §1 · p. 2
  • A safety-relevant change becomes evolutionarily relevant when it is retained, reused, or passed to a successor, and its effects may outlast the initiating input.Such changes can reside in memory or skills, model parameters, feedback, computational substrate, or update mechanisms.Source: §1 · p. 2
  • The paper identifies five domains that carry or mediate change: persistent agent state, model state, evaluation and environmental feedback, computational substrate, and meta-level update mechanisms.It frames the inquiry around how long changes remain influential, which system parts they reach, and how much of the improvement loop is closed within the system.Source: §1 · p. 3
  • Evaluation units span states, accepted updates, trajectories, and lineages, linked to requirements for modification authority, evidence, provenance, and recovery.Lineages trace propagation across descendants or interacting systems.Source: §1 · p. 3

2 Foundations of Recursive Self-Improving AI and Evolutionary Safety

This section establishes foundations for evolutionary safety by defining persistent self-improvement and its structural forms, and explaining when failures become evolutionary. It examines how safety properties persist, weaken, and recover through persistent state, selection and evaluation, and the inheritance of risks across updates and descendant lineages.

2 Foundations of Recursive Self-Improving AI and Evolutionary Safety

  • Safety is consequential beyond one interaction when changes persist, are selected or inherited, and alter the starting conditions for future behavior.As persistence grows, mutable scope can extend from outputs to memories, tools, parameters, evaluators, and the update process itself.Source: §2 · p. 3

2.1 What Counts as Persistent Self-Improvement?

  • A persistent self-improver helps generate experience, propose or select changes, or modify its evaluator or updater, then carries results into later behavior.An episode-only response revision is local improvement; reusable changes to memory, skills, tools, workflows, or models are persistent adaptation, while changing the process that generates or admits changes is recursive development.Source: §2.1 · p. 4
  • An accepted improvement can still weaken safety, and a coding agent’s orchestration-code edits showed persistent benchmark gains without addressing authorization, provenance, or recovery.The example demonstrates a route to persistent change, not safeguards for governing that change.Source: §2.1 · pp. 4–5

2.2 From Local Improvement to Recursive Development

  • Persistence, mutable scope, and loop closure distinguish local improvement, persistent adaptation, and recursive development structurally, not by intelligence.These dimensions track how long updates matter, what can change, and how much of the improvement loop the system controls.Source: §2.2 · p. 5

2.3 Why Safety Becomes Evolutionary

  • A safety failure becomes evolutionary when its effects persist, influence later selection, or propagate to later states or successors.A variation may introduce a trade-off; selection can retain it, artifacts or descendants can carry it forward, and evaluator or updater changes can alter the selection pressure itself.Source: §2.3 · p. 5

2.4 Manifestations of Safety Degradation

  • The six degradation manifestations are intent drift, error accumulation, experience contamination, safety-property erosion, evaluator drift, and risk inheritance or propagation.They form a diagnostic ladder tracing how retained changes are selected and where their effects reappear.Source: §2.4 · p. 6
  • The diagnostic order can be skipped or branch across artifacts, with failures feeding back into evaluators; intent drift, error amplification, and contaminated experience illustrate early mechanisms.Intent drift changes operational priorities, small errors gain influence when later updates build on them, and erroneous, biased, or adversarial information can enter learning material.Source: §2.4 · pp. 6–7
  • A safety risk can persist when an encountered input becomes retained material that shapes later behavior, even after its source is removed.The key transition is from exposure to persistent influence.Source: §2.4 · p. 7
  • Successive updates can erode a reliable safety property even as task performance improves, and a defense that works at one checkpoint may fail after mixed updates.Capability gain and safety retention therefore require separate measurement.Source: §2.4 · p. 7
  • Evaluation both admits candidates and sets the standard for judging successors, allowing an inherited omission to create a feedback loop.A judge trained on accepted candidates may reproduce the same omission.Source: §2.4 · p. 7
  • Unsafe tendencies can propagate through experience, skills, generated traces, merged models, or interacting agents, so repair requires lineage records and independent descendant testing.Deleting the initiating record alone is insufficient because each transfer can alter how a risk is represented, activated, and detected.Source: §2.4 · p. 7
  • Diagnosing degradation asks which safety property changed, what sustained the change, which later states express it, and whether descendants or interacting systems inherited it.These questions help distinguish persistence from repeated exposure, selection from transmission, and local repair from effective recovery.Source: §2.4 · pp. 7–8

2.5 Defining Evolutionary Safety

  • Evolutionary Safety examines how safety properties are preserved, weakened, or restored as systems retain and build on their own improvements.It tracks how selection, persistent state, and inheritance shape those properties across updates, interacting systems, and descendant lineages.Source: §2.5 · p. 8
  • Evolutionary analysis distinguishes state, accepted-update, trajectory, and lineage assessments, while separating safety failures from distributional failures or losses of useful capability.Risk and control should be reported as separate, property-specific readouts with application-specific thresholds and operating conditions.Source: §2.5 · p. 8

3 A Taxonomy of Evolutionary Safety

The taxonomy organizes evolutionary safety around the carriers of change and the routes through which changes affect future systems. It spans agent and model state, evaluation feedback, reused execution substrates, and the mechanisms and interactions shaping future improvement.

3 A Taxonomy of Evolutionary Safety

  • The taxonomy tracks changes by carrier and route across persistent agent artifacts, model parameters, evaluator and environmental feedback, computational substrates, and meta-level updates.Application area, capability level, and harm severity cut across these domains.Source: §3 · p. 8

3.1 Persistent Agent State

  • When observations become reusable rules or executable skills, they can outlive their source context and extend unsafe behavior into later actions.A memory write can be summarized into a skill, retrieved later, and used in a tool action; deleting the source does not necessarily remove the derived artifact.Source: §3.1 · p. 9
  • Selective repair is difficult because provenance can disappear during rewriting, while deleting every derived artifact may discard useful capability.A locally successful strategy may also generalize poorly when its reflection omits the conditions under which a shortcut was acceptable [104] [138].Source: §3.1 · pp. 9–10

3.2 Model State

  • Self-training, distillation, and merging can improve capability while changing earlier learned safety properties; distributed learned tendencies may affect behavior beyond explicit memory, with effects emerging in new tasks or descendants.In the studied settings, trained backdoor behavior survived subsequent safety training [54].Source: §3.2 · p. 10
  • Repeated training on model outputs can distort learned distributions, and retaining real data changes the conditions under which collapse occurs [2] [40].Distributional stability and alignment preservation are separate properties that should be measured separately across successive self-training and self-play updates [51] [137].Source: §3.2 · p. 10

3.3 Evaluation and Environmental Feedback

  • Self-generated tasks and revisable judges can make shortcuts increasingly attractive while weakening the tests used to assess candidates, complicating comparisons across rounds.The safety challenge is to distinguish changes in candidate experience from changes in selection criteria.Source: §3.3 · pp. 11–12
  • Task selection shifts future experience and can hide favored-strategy failures; compare candidate generation with a fixed evaluator and selection by replaying candidates under alternative evaluators.Source: §3.3 · p. 12

3.4 Computational Substrate

  • Changes to the execution substrate can weaken controls, and a defect that passes functional checks may spread through reuse into descendant designs.Source: §3.4 · p. 12

3.5 Meta-level Update Mechanisms and Co-evolution

  • Updater revision can threaten surviving constraints, while co-evolution reinforces local strategies; both challenge evaluation and control as future candidates evolve.Source: §3.5 · p. 12

3.5.1 Changing the Updater.

  • Changing the procedure that admits updates can bypass security tests and outgrow prior evaluations; comparing frozen and mutable updaters under common external evaluation isolates this effect.Source: §3.5.1 · p. 13

3.5.2 Reciprocal Adaptation.

  • Reciprocal adaptation can alter future improvement even with a frozen updater: strategies or unsafe conventions may spread, while role-based evaluation and perturbations reveal hidden vulnerabilities.Source: §3.5.2 · p. 13

4 Discovering and Evaluating Evolutionary Risk

Evolutionary risk evaluation should track safety across states, updates, trajectories, and lineages, using controlled comparisons and interventions to assess whether changes are attributable to evolution and to uncover known and emerging failures. It should also account for changing evaluators and reference anchors, and dependencies that may compromise evaluator independence.

4 Discovering and Evaluating Evolutionary Risk

  • Evaluate states, individual updates, multi-update trajectories, and lineages to capture point-in-time risk, change, accumulation, and propagation.Use controlled comparisons and interventions to assess whether observed changes are attributable to evolution.Source: §4 · p. 13

4.1 Evaluation Units and Measurements

  • Track trajectories as well as endpoints, and test persistence, propagation, recovery, and population-level risk separately.Trajectories can expose gradual degradation, delayed threshold crossings, or recurrence after repair; lineage measures include propagation reach, depth, and delay. Population averages can conceal an unsafe subgroup, so report risk distributions and affected roles.Source: §4.1 · p. 14

4.2 Known-Risk Detection and Emerging-Risk Discovery

  • For known risks, repeat fixed held-out tests after updates; for emerging risks, use adaptive discovery beyond the original test surface.Validate discoveries by preserving trigger conditions and evaluator versions, reproducing failures on saved states, and testing nearby cases; retain validated cases as regression tests and report discovery yield separately from longitudinal risk.Source: §4.2 · pp. 14–15

4.3 Evolution Protocols and Causal Attribution

  • Evolution protocols should include benign adaptation, misspecified rewards, poisoned experience, and adaptive peers, while source removal tests whether risk persists without repeated exposure.Compare matched initial configurations that differ in one declared intervention, holding the external evaluation reference and specified conditions fixed; resume adaptation after repair or rollback to test recurrence.Source: §4.3 · p. 15
  • To attribute change to updating, compare regimes that begin from same configuration and differ in one declared intervention; assess selection shifts within a common candidate pool, including rejected candidates.Source: §4.3 · p. 16

4.4 Evaluation Validity under Evolving Evaluators

  • Evaluator revision can change which candidates survive and how systems are scored, so compare saved states across evaluator versions and successive states under one evaluator.Use a fixed external reference for longitudinal comparison and an adaptive evaluator to discover failures; without overlapping comparisons, evaluator and system changes cannot be separated.Source: §4.4 · p. 16
  • Fixed references can age, so monitor their coverage with independent tests, retain earlier anchor versions, and rescore overlapping states before comparing versions.Source: §4.4 · p. 16
  • Adaptive evaluators expand failure discovery, but their changing task mix should remain separate from older scores [116] [134].Source: §4.4 · p. 16
  • Evaluator independence requires more than a different prompt: shared data or model families can correlate errors, and systems may behave differently when evaluation is expected [44] [83].Held-out tasks and isolated execution reduce some dependencies; restricted evaluator access and separate authorization address others [19]. Reports should state which dependencies were controlled and which remain.Source: §4.4 · pp. 16–17

5 Evolutionary Safety Across Application Domains

Four application settings show how retained experience or changes can alter later behavior and widen the consequences of a local failure. Each requires evaluation and recovery suited to its setting.

5 Evolutionary Safety Across Application Domains

  • The four scenarios connect persistence and inheritance to distinct application settings and evaluation or recovery needs.They cover critical infrastructure, automated AI R&D, embodied systems, and scientific discovery.Source: §5 · p. 17
  • A spurious experimental result reused to choose hypotheses can appear better supported when later evidence depends on the same error.Evaluation should separate independent replication from reused evidence and test correction across dependent plans; rolling back a record cannot undo an experiment already performed. Current AI-scientist systems are not yet built for unsupervised discovery.Source: §5 · p. 17
  • A grid controller adapted for ordinary loads may improve short-term performance while reducing its contingency margin.Evaluation follows violations and load shedding across updates; recovery must stabilize the physical system because software rollback cannot reverse a cascade.Source: §5 · p. 17
  • A flawed repair becomes a development-wide risk when it enters a harness used to build successor systems.Recovery must revalidate dependent artifacts, and increased experiment throughput is useful only when verification can assess the added changes.Source: §5 · p. 17
  • Reusing an unchanged robot skill can increase exposure when it crosses a boundary that matters in a new setting.Evaluation should measure violations, near misses, and transfer; recovery must address both software reuse and physical consequences.Source: §5 · p. 17

6 Discussion: Control and Governance Implications

Governance controls belong inside the evolution loop: constrain proposed changes, verify them independently, and retain traceability and recovery paths. Their strength should scale with a change’s scope, persistence, and downstream reach.

6 Discussion: Control and Governance Implications

  • Declare what a system may change and which authority can expand that scope, with enforcement outside the editable component.Provenance checks also bound which experience and generated artifacts may influence later adaptation.Source: §6 · p. 17
  • Keep candidates separate from the active lineage until sandboxed or shadow tests compare their performance and safety.A gate provides evidence but should not silently grant the candidate more authority; tests should include dependencies and plausible reuse.Source: §6 · p. 18
  • Separate the ability to propose a change from the authority to commit it, using independent verification and external commit control.A different model alone is insufficient if it shares the same blind spot, and evaluator or update-procedure changes require approval from an authority they cannot replace.Source: §6 · p. 18
  • Link accepted changes to their source, test evidence, verifier, permissions, and downstream dependencies so affected descendants can be quarantined or revalidated.Recovery should retain useful unaffected state where possible, report uncertainty when records are incomplete, and treat software restoration and containment of physical harm as separate obligations.Source: §6 · p. 18
  • Broader and more persistent changes require stronger evidence, authorization, traceability, and recovery mechanisms.The goal is not to prevent change, but to prevent consequential changes from becoming persistent, selected, or inherited without adequate evidence and a viable recovery path.Source: §6 · p. 18

7 Open Problems

The open problems concern preserving safety across changing systems and discovering risks that may be delayed, latent, or propagated. They also include recovery and oversight when evidence is incomplete or systems interact.

7 Open Problems

  • Safety properties may need to survive heterogeneous sequences of updates, including changes to tasks, tools, and permissions.Proposed tests mix memory writes, model training, and evaluator revisions; delayed attribution needs known injected causes to measure replay and rollback against delay and lineage depth.Source: §4.1 · p. 14, §7 · p. 18
  • Changing evaluators and partial observability make it difficult to distinguish undiscovered failures from failures hidden by unavailable evidence.The independence needed between generator, actor, and judge remains unresolved.Source: §7 · pp. 18–19
  • A social heuristic learned from repeated interactions may treat familiarity as authorization and later affect behavior despite unchanged permissions.Experiments can vary relationship history and oversight while holding tasks and permissions fixed, then compare frozen and updating memory and follow correction through later versions.Source: §7 · p. 19
  • Evidence strongest for individual memory or model pathways does not establish how risks propagate across combined artifacts or co-evolving populations.Benchmarks need known derivation histories and interventions that distinguish transmission from shared exposure; isolated-agent results cannot establish collective safety.Source: §7 · p. 19
  • Recovery can restore an old score yet still remove useful capability, miss derived artifacts, or leave physical harm untreated.Evaluation should jointly report useful discoveries, unsafe exposure, containment failures, and recurrence after repair.Source: §7 · p. 19

8 Conclusion

The conclusion frames safety as a property to track through persistent improvement and successor production, not only at one system state. It presents a framework for characterizing risks, evaluating them across evolutionary units, and governing changes.

8 Conclusion

  • Evolutionary Safety studies how safety changes as AI systems improve, persist, and produce successors.Changes become evolutionarily consequential when they are retained, selected, inherited, and propagated.Source: §8 · p. 19
  • The framework locates risk mechanisms across agent and model state, feedback, computational substrate, and meta-level update mechanisms with co-evolution.It characterizes manifestations from intent drift and error accumulation to evaluator drift and risk propagation.Source: §8 · p. 19
  • The paper develops evaluation across states, updates, trajectories, and lineages, alongside governance through modification boundaries, independent evaluation and authorization, provenance, and recovery.As AI shapes its own development, safety requires understanding and governing how systems continue to change.Source: §8 · pp. 19–20

Related papers

Loading 2609.31186v1…