PaperlayerView exact revision on arXiv

Source-linked AI summary

Evolutionary Computation for Trustworthy AI: From Attacks and Defenses to Self-Evolving Era

Junhao Dong, Chenkai Wang, Xuanhui Lin, Mingrong Gong, Siyu Wang, Yuqing Wen, Jiao Liu, Catherine Huang, Gary G. Yen, Xin Yao, Yew-Soon Ong

arXiv:2610.02996v1cs.NE

TL;DR

Trustworthy AI research, evolutionary computation, and self-evolving systems have largely been reviewed separately, despite shared challenges in black-box and structured search. This survey connects them across attacks, defenses, and trustworthy self-evolution, and highlights how persistent updates make adaptation itself a trustworthiness concern.

Abstract

from arXiv · show

As Artificial Intelligence (AI) has evolved from task-specific models to foundation models and agents, the scope of trustworthy AI has expanded from model-level robustness to the reliability and safety of broader AI systems. This evolution has also expanded the attack surface from individual models to broader system-level interactions, including tool use, context, and interaction trajectories with dynamic environments. As a result, maintaining reliable and safe behavior under changing or deliberately manipulated conditions has become increasingly challenging. The search for effective attacks and defenses often relies on black-box feedback to navigate discrete choices among words, actions, system components, or their combinations. Multiple objectives and expensive candidate evaluations further limit what can be explored. Evolutionary Computation (EC), with its population-based, gradient-free search and flexible variation and selection mechanisms, is well suited to these settings. This survey reviews how EC has been applied to trustworthy AI across three directions: evolutionary attacks, evolutionary defenses, and trustworthy self-evolving AI systems. Unlike prior reviews that treat trustworthy AI, EC, and self-evolving systems largely separately, we connect these lines through a common evolutionary perspective. For self-evolving AI, we examine how trustworthiness governs the generation and retention of updates that shape subsequent adaptation. We further synthesize evaluation methods and benchmark resources from both trustworthiness and evolutionary-search perspectives. Finally, we discuss key challenges and future research directions toward more effective and reliable use of EC in trustworthy AI.

I. Introduction

The survey links evolutionary attacks and defenses with trustworthy self-evolving AI, addressing a fragmented literature and the challenges of searching complex, changing systems. It frames persistent updates as a distinct trustworthiness concern: not only which candidate is selected, but which changes are allowed to endure.

I. Introduction

  • Trustworthy search faces black-box feedback, discrete or structured choices, competing criteria, a need for behavioral diversity, and changing attack–defense conditions.Attack strategies can alter which defenses remain effective, and defensive updates can change which attacks succeed in later evaluations.Source: §I · p. 1, attackers, defenders, and agents may… · p. 2
  • EC's gradient-free search supports black-box and structured optimization, while multiobjective selection preserves alternatives across competing trustworthiness criteria.The survey also identifies surrogate-assisted optimization and bi-fidelity search as ways to reduce costly evaluations.Source: attackers, defenders, and agents may… · p. 2
  • Diversity-preserving and quality-diversity search retain behaviorally distinct candidates that can expose different vulnerabilities or system failure modes.Uncertain quality-diversity methods can improve archive construction under noisy or uncertain evaluation.Source: attackers, defenders, and agents may… · p. 2
  • Co-evolution models reciprocal adaptation between attackers and defenders, while constraints, safety-aware selection, and validation regulate persistent updates.Retained jailbreaks may guide later guardrail updates, which can reshape the attacks that remain effective.Source: attackers, defenders, and agents may… · p. 2
  • The survey unifies evolutionary attacks, evolutionary defenses, and trustworthy self-evolving AI, while reviewing evaluation, benchmarks, and open challenges.Its scope includes search efficiency, generalization, evaluation reliability, and safe persistent adaptation.Source: attackers, defenders, and agents may… · pp. 2–3
  • Self-evolving AI shifts the concern from choosing a good candidate to controlling which accepted changes persist and shape later system versions.The survey also identifies search efficiency, generalization, evaluation reliability, and safe persistent adaptation as open challenges.Source: attackers, defenders, and agents may… · p. 3

A. Review Scope

The survey defines its scope through three roles for evolution and treats evaluation as a cross-cutting concern. It positions its contribution as connecting trustworthy AI, evolutionary computation, and persistent system evolution.

A. Review Scope

  • Self-evolution is distinguished by inheritance: an accepted update qualifies when it affects the state available to subsequent system versions.The reviewed systems repeatedly generate, evaluate, retain, and reuse updates to memories, skills, workflows, safeguards, or learning data.Source: A. Review Scope · p. 4
  • The survey covers evolutionary attacks, evolutionary defenses, and trustworthy self-evolving AI, with evaluation spanning trustworthiness outcomes and evolutionary-search performance.These roles correspond to compromising, strengthening, or adapting persistent system state.Source: A. Review Scope · pp. 3–4

B. Comparison with Existing Surveys

  • The survey connects three research lines by relating EC and persistent system evolution to trustworthy AI.It distinguishes trustworthiness through evolution, such as vulnerability discovery or mitigation, from trustworthiness of inherited evolution over time.Source: B. Comparison with Existing Surveys · pp. 4–5
  • Existing surveys generally emphasize either trustworthy AI or EC methodology, with limited coverage of trustworthy self-evolution across predictive, generative, and agentic systems.Figure 3 summarizes the development of these directions and presents a 3S framework: constrained exploration, selection-guided refinement, and attack–defense co-evolution.Source: B. Comparison with Existing Surveys · p. 5, Figure 3 · p. 5

III. Preliminaries

The preliminaries establish evolutionary search as the shared optimization process used throughout the survey, then extend the toolbox with complementary biologically inspired paradigms. They also frame adversarial and self-evolving AI in terms of search and persistent state updates.

III. Preliminaries

  • The preliminaries formulate adversarial machine learning as minimax optimization and describe EC attacks, defenses, joint attack–defense evolution, and persistent updates to system state.The formulations connect accepted updates to subsequent changes in self-evolving systems.Source: §III · p. 5

A. Evolutionary Search Process

  • Evolutionary search initializes a population, selects parents by fitness, varies them into offspring, and selects survivors for the next generation.When fitness is vector-valued, selection reflects trade-offs among multiple objectives.Source: A. Evolutionary Search Process · p. 5
  • Memetic computing combines population-based search with local improvement or problem-specific learning while retaining candidate diversity.The combination can use available learning signals while preserving alternatives and trade-offs.Source: A. Evolutionary Search Process · pp. 5–6
  • Artificial immune systems use mechanisms such as self/non-self discrimination, adaptive defense, and immune memory for tasks including anomaly and intrusion detection.The survey presents AIS as complementary to explicit evaluation of trustworthiness properties such as safety, robustness, fairness, and reliability.Source: A. Evolutionary Search Process · p. 6

B. Attack and Defense via Evolutionary Search

Evolutionary computation searches attacks and defenses as populations of candidates, including under black-box feedback and constraints. Coupled co-evolution lets attacks and defenses adapt to the current state of their counterparts.

B. Attack and Defense via Evolutionary Search

  • An attack can maximize model loss within an admissible perturbation set, while adversarial training minimizes expected loss under worst-case perturbations.EC can be applied separately to the attack side, the defense side, or jointly to both.Source: B. Attack and Defense via Evolutionary… · p. 6
  • Population-based, gradient-free search can evolve attack and defense candidates beyond input perturbations or model parameters to prompts, policies, agent structures, and other system components.Selection, variation, replacement, diversity maintenance, and archive management can shape populations without requiring explicit analytical objectives.Source: B. Attack and Defense via Evolutionary… · p. 6
  • Multiobjective and quality-diversity methods can preserve complementary candidates across trade-offs such as attack effectiveness, stealth, validity, cost, robustness, utility, and efficiency.Quality-diversity methods maintain behaviorally distinct candidates across niches or archives.Source: B. Attack and Defense via Evolutionary… · p. 6
  • In co-evolution, attack candidates are evaluated against the current defense population, while defense candidates are evaluated against the current attack population.The reciprocal updates allow stronger attacks to guide defense evolution and improved defenses to create selection pressure for subsequent attack search.Source: B. Attack and Defense via Evolutionary… · p. 6

C. State Updates in Self-Evolving AI

Self-evolving AI retains accepted updates as part of system state, so they can affect later behavior and updates. The section frames trustworthy state updates around admission, state modification, and evaluation of both task outcomes and search efficiency.

C. State Updates in Self-Evolving AI

  • Unlike one-shot optimization, accepted changes persist in system state and influence later behavior or subsequent updates.This persistence raises concerns that unsafe memories, brittle workflows, unreliable skills, or adversarial strategies may accumulate and propagate failures across versions.Source: C. State Updates in Self-Evolving AI · pp. 6–7
  • A state transition admits or rejects a candidate update, and an accepted update modifies the current state inherited by later iterations.The update-admission decision is G(ut; st), and Φ(st, ut) specifies how an accepted update modifies the state.Source: C. State Updates in Self-Evolving AI · p. 7
  • Evolutionary search can maintain candidate updates as a population, using variation and selection while trustworthiness constrains admissible updates, admission, or feedback.Multiobjective selection, diversity maintenance, and archives can preserve candidates with different trade-offs or behaviors.Source: C. State Updates in Self-Evolving AI · p. 7

IV. Evaluation and Benchmarking

  • Evaluation measures task-level effectiveness and search-level efficiency and reliability, with benchmarking specifying tasks, metrics, and computational budgets.Search-level measures include diversity, evaluation cost, and convergence.Source: §IV · p. 7

A. Trustworthiness Evaluation

The evaluation framework combines task-level trustworthiness measures with evolutionary-search measures and benchmark settings. It covers robustness, safety, privacy, task utility, optimization performance, diversity, budget, convergence, and repeated-run analysis.

A. Trustworthiness Evaluation

  • Trustworthiness evaluation measures robustness under a threat model, safety and privacy outcomes, and preservation of task performance alongside attack success or other trustworthiness metrics.Examples include attack success, perturbation magnitude, query cost, jailbreak success, behavioral coverage, inversion success, clean accuracy, task success, and refusal rates.Source: 1) Robustness · p. 7, 2) Safety and Privacy · p. 7, 3) Task Performance Preservation · p. 7

B. Evolutionary Search Evaluation

  • Evolutionary-search evaluation uses best or final fitness for single-objective methods, hypervolume for multiobjective methods, and fitness, archive coverage, or QD score for quality-diversity methods.Search cost can be measured by model queries, judge evaluations, architecture evaluations, tool calls, or environment rollouts; results across runs may use medians, interquartile ranges, and confidence intervals.Source: 1) Optimization Performance and… · p. 7, 2) Budget and Convergence · p. 7

C. Benchmarking

  • Trustworthiness benchmarks provide task-level metrics across model robustness, language and multimodal safety, agent execution, risk awareness, moral judgments, and social bias.The cited resources include RobustBench, JailbreakBench, Agent-SafetyBench, R-Judge, ETHICS, and BBQ.Source: C. Benchmarking · pp. 7–8
  • EC benchmarks assess evaluation cost, target attainment, convergence, and repeated-run performance, while comparisons should specify both task and search settings.The task setting includes the target system, threat model, evaluator, and task metrics; the search setting includes cost unit, budget, stopping condition, and independent runs.Source: C. Benchmarking · pp. 7–8

V. Evolutionary Attacks

Evolutionary attacks search over candidates suited to predictive, generative, and agentic systems, with constraints such as semantic preservation, stealthiness, functionality, and query efficiency. The surveyed methods extend from perturbations and triggers to prompts, external documents, and model-stealing queries.

V. Evolutionary Attacks

  • Evolutionary attacks encode perturbations, backdoor triggers, prompts, adversarial documents, or agent prompts as candidates and optimize them through selection and variation under system feedback.The taxonomy organizes attacks by target system and then by attack surface, objective, or search object.Source: §V · p. 8, Figure 4 · p. 8

A. Attacks on Predictive Models

B. Attacks on Generative Models

  • Generative-model jailbreak methods evolve structured prompts, with fitness balancing bypass success against qualities such as readability, semantic preservation, stealthiness, or transferability.AutoDAN uses hierarchical genetic search over prompt structure; other methods include multiobjective search and language-model-based variation [54], [55].Source: Figure 4 · p. 8, 1) Evolutionary Jailbreak · p. 10
  • Generative-model attacks also evolve external documents to poison RAG pipelines and prompts that produce informative query data for model stealing.Methods shift search from the user prompt to retrieved knowledge passages or model-extraction queries.Source: 2) Evolutionary RAG Attacks and Model… · p. 10

C. Attacks on Agentic Systems

In agentic systems, evolutionary attacks target the effects of injected content and can extend to trajectory- and environment-level feedback. Across the supplied defense passages, EC also searches for training examples that improve robustness or safety.

C. Attacks on Agentic Systems

  • Agentic attacks evolve interface injections, then extend search toward execution trajectories and adversarial environment states; persistent memory, multi-agent interaction, and embodied execution remain underexplored.T-MAP uses trajectory-aware MAP-Elites, while OpenART searches adversarial environment states.Source: 1) Evolutionary Injection and… · p. 10

D. Discussion

  • Attack performance depends on representation, fitness design, feedback quality, and query budget, while varying several factors together can obscure EC’s contribution.The survey calls for more controlled comparisons with alternative black-box search methods.Source: D. Discussion · p. 10

VI. Evolutionary Defenses

  • Evolutionary defenses use an outer-loop search over training data, model design, training strategies, or deployment safeguards, evaluated for robustness, safety, and normal task performance.The evolutionary object shifts from training examples to structures, learning strategies, and deployed behaviors.Source: §VI · p. 11

A. Training Data

  • Defensive data-level search evolves examples that expose weaknesses or safety-relevant behaviors, then feeds selected examples into training or alignment.ER-APT illustrates this workflow by refining adversarial examples for robust prompt learning.Source: A. Training Data · p. 11
  • Adversarial-training methods evolve examples or their proportions, with EMO-GAT, ER-APT [66], and IGAff [67] optimizing training data through selection, transformation, or composition.ER-APT combines gradient-based generation with genetic selection, mutation, and crossover.Source: 1) Evolutionary Adversarial Training · p. 11
  • Safety-data methods evolve instructions that expose failures or unnecessary refusals, then reuse them for alignment, including approaches for language and vision-language-action systems.Rainbow Teaming uses quality-diversity search to generate prompts for fine-tuning against unseen attacks.Source: 2) Safety and Alignment Data Generation · p. 11

B. Model Design

For model design, EC searches robustness-aware architectures and model configurations. Training optimization instead evolves objectives, training strategies, alignment parameters, or policies under safety and robustness criteria.

B. Model Design

  • Model-design search evolves architectures and configurations using robustness-aware fitness, including trade-offs with accuracy, model size, backdoor mitigation, ensemble diversity, or fairness.MORAS trades predictive accuracy against robustness to multiple attacks; ER-APT [66] refines adversarial examples for robust prompt learning.Source: 1) Robust Architecture Search · pp. 11–12, B. Model Design · p. 11, Figure 5 · p. 12

C. Training Optimization

  • Training optimization evolves loss functions, multi-objective training configurations, adversarial-training strategies, or alignment adapters to shape robust or safe behavior.EvoPref explores LoRA adapters against helpfulness, harmlessness, and honesty objectives.Source: 1) Objective and Training Strategy… · p. 12, C. Training Optimization · p. 12
  • Safe and robust policy search makes policy constraints part of optimization or selection, combining evolution with statistical checking or verification.LyEvO combines constrained evolutionary optimization with statistical verification and Lyapunov analysis for safe, robust sim-to-real policy learning.Source: 2) Safe and Robust Policy Search · pp. 12–13

D. Deployment Safeguards

Evolutionary defenses range from input-specific inference-time adaptation to post-training backdoor mitigation. In self-evolving systems, trustworthiness also governs which changes are admissible and which updates persist into later adaptation.

D. Deployment Safeguards

E. Discussion

  • EC typically acts as an outer-loop optimizer around conventional defense mechanisms, with newer approaches increasingly adapting within training or execution.The evolutionary object depends on where search enters the defense pipeline and what counts as a candidate solution.Source: 2) Post-Training Backdoor Mitigation · p. 13

VII. Trustworthy Self-Evolving AI

  • Self-evolving AI retains accepted changes that shape later behavior or updates, so trustworthiness can intervene through constrained search, update selection, or attack–defense co-evolution.Unsafe memories, brittle workflows, unreliable skills, or adversarial strategies may accumulate and lead to safety drift or propagating failures.Source: §VII · p. 13

A. Search-Space-Constrained Exploration

  • Search-space-constrained methods restrict candidate updates through structural, semantic, or behavioral requirements, including typed workflows and formal verification.MermaidFlow [32] limits variations to well-formed, executable descendants; SEVerA [86] admits programs to later learning only after specified behavioral properties are verified.Source: A. Search-Space-Constrained Exploration · p. 13

B. Selection-Guided Refinement

  • Selection-guided refinement allows candidate changes but uses safety-aware objectives, validation, or verification to decide what is retained and reused in later evolution.HarnessBank [34] screens proposed harness changes before adding them to a reusable archive; other examples govern memories, skills, or localized safety updates.Source: B. Selection-Guided Refinement · pp. 13–14

C. Strategic Attack–Defense Co-Evolution

Strategic attack–defense co-evolution uses feedback between adaptive attackers and defenders rather than treating adversarial evaluation as fixed. The survey connects this process to persistent update state and identifies future challenges involving search scale, diversity, generalization, and benchmarking.

C. Strategic Attack–Defense Co-Evolution

  • In attack–defense co-evolution, new attacks reshape defense training and improved defenses alter later attack effectiveness and selection pressure.ROMANCE, CEMMA [92], and Evo-MARL [19] illustrate reciprocal adaptation; DARWIN [35] and EvoSafety [93] retain attack and defense state across rounds.Source: C. Strategic Attack–Defense Co-Evolution · p. 14

D. Discussion

  • Persistent evolutionary state links current selection decisions to the search and safety of later system versions, making robustness a property of the update process as well as each solution.Reusable populations or external stores can carry evolutionary state across rounds.Source: C. Strategic Attack–Defense Co-Evolution · p. 14

VIII. Open Challenges and Future Research Directions

  • The survey identifies four open challenge areas: multi-component optimization, multi-solution defense, multi-task generalization, and multi-objective benchmarking.Fig. 7 maps these challenges to future research directions.Source: §VIII · p. 14, Figure 7 · p. 15

A. Multi-Component Search under Limited Budgets

  • Jointly evolving agent components enlarges the search space and evaluation budget, while component costs vary; selective search could prioritize inexpensive, high-impact parts and leave costly components unchanged unless needed.Interactions among agents can add coupled safety and security risks.Source: A. Multi-Component Search under… · p. 15

B. Multi-Solution Defense for Diverse Threats

The section calls for complementary defenses, cross-task evaluation of self-evolving updates, and benchmarks that jointly assess trustworthiness and evolutionary-search performance.

B. Multi-Solution Defense for Diverse Threats

  • A single defense may fail across attack types or requirements, motivating complementary solutions combined through ensembles, voting, or context-dependent selection.Different solutions can target different attacks, data conditions, or safety requirements.Source: B. Multi-Solution Defense for Diverse… · p. 15

C. Multi-Task Generalization in Self-Evolving AI

  • Updates should be evaluated across tasks because gains on one task may not transfer and can reduce previously learned capabilities.Cross-task generalization with capability retention relates to the stability–plasticity trade-off in continual learning.Source: C. Multi-Task Generalization in… · p. 15

D. Multi-Objective Benchmarking for EC and Trustworthy AI

  • Unified benchmarks should assess robustness, safety, task performance, search cost, and convergence under the same evaluation budget.Weighted scores may summarize these dimensions, but individual metrics and trade-offs should also be reported.Source: D. Multi-Objective Benchmarking for EC… · pp. 15–16

IX. Conclusion

  • The survey organizes EC for trustworthy AI around what is evolved, how search is performed, and how outcomes are evaluated.It covers attacks, defenses, and self-evolving AI, including constraints on evolution, update retention, and attack–defense co-evolution.Source: §IX · p. 16

Related papers

Loading 2610.02996v1…