ReSI: Recursive Safety Improvement toward Resistant and Resilient AI

1. Introduction

• Because safety alignment for one checkpoint may not carry over to successors, each successive checkpoint requires renewed alignment.Evolving red-teaming methods continue to expose vulnerabilities, including multi-turn attack weaknesses in current state-of-the-art models.Source: §1 · p. 3
• ReSI searches over attack data, alignment methods, data mixtures, and training configurations because the effective recipe depends on the current model and newly identified attacks.The paper motivates experimental exploration rather than a fixed procedure for selecting and combining candidate methods.Source: §1 · p. 3
• Each ReSI round red-teams the target, screens attack-source and alignment-method combinations, then refines shortlisted recipes through full-scale training and feedback.A Pareto gate requires safety improvement while preserving instruction following and limiting over-refusal; the best eligible checkpoint becomes the next target.Source: §1 · p. 3, Figure 2 · p. 4
• Recursion is operationally bounded: the search procedure and evaluation protocol remain fixed across rounds, while only the target model changes.The framework's red-teaming and alignment-method pools can be extended with new attacks and learning approaches.Source: §1 · p. 4
• On X-Teaming, ReSI-trained models had ASRs of 16.98–43.40% versus 56.69–84.85% for evaluated frontier models.Across the safety panel, their ASRs also fell within or below corresponding frontier-model ranges, while general capabilities and benign compliance were largely preserved.Source: §1 · p. 4
• ReSI improves in-distribution resistance and resilience to unseen attacks, outperforming A3 and matching or surpassing MAGIC in nearly all available resistance comparisons.All four ReSI-trained models substantially outperform frontier-model comparators on X-Teaming, an adaptive multi-turn attack outside the training red-teaming pool.Source: §1 · pp. 4–5

2. The ReSI Framework

• ReSI repeats red-teaming, recipe development, and model updating across checkpoints, with each accepted update becoming the next target model.The loop terminates at its round limit or when no candidate passes the Pareto gate, retaining the current model in the latter case.Source: §2 · p. 5, §2.4 · p. 10

2.1. Stage 1: Red-Teaming Assessment

• Red-team methods attack the current target using deployment-risk requests, and input–response records that elicit unsafe responses form training signals for recipe search.The attack and safety-data pools can incorporate new methods and data as risks evolve.Source: §2.1 · p. 5

2.2. Joint Recipe Search Space

• Each recipe jointly specifies attack-source contributions, an alignment method, a mixture of current, replay, and general-task data, and training hyperparameters.The data mixture balances learning from identified vulnerabilities with retaining existing safety behavior and general capabilities.Source: §2.2 · pp. 6–7

2.3. Stage 2: Pilot Screening

• Pilot trials compare attack-source and alignment-method combinations under low budgets, then shortlist recipes with the largest positive safety gains.Conditional scaling probes increase attack counts for recipes with modest gains and more available data before ranking them alongside initial pilots.Source: §2.3 · pp. 7–8

2.4. Stage 3: Full-Scale Refinement and Model Update

• Full-scale trials refine data mixtures and training parameters using gate feedback, then select the eligible update with the largest observed safety gain.The gate requires safety improvement while keeping benign compliance and instruction following within tolerance of the initial model; refinements can expand attack sources, add harmful–benign pairs, increase general-task replay, or continue eligible SFT recipes with safety GRPO.Source: §2.4 · pp. 8–10

3. Experimental Setup

• WildJailbreak assesses in-distribution safety, HarmBench existing-defense retention, H-CoT and X-Teaming out-of-distribution attacks, XS-Test benign over-refusal, and three benchmarks general capabilities.GPQA-Diamond, MMLU-Pro, and IFEval assess reasoning, knowledge proficiency, and instruction following, respectively.Source: §3 · pp. 10–11
• Attack success rate measures successful attacks, while defense success rate is 1 − ASR; benign compliance, benchmark accuracy, and loose instruction-following accuracy measure the other reported axes.Lower ASR indicates stronger safety; higher benign compliance indicates less over-refusal.Source: §3 · p. 11
• Safety-alignment comparisons use A3 and MAGIC, while four leading frontier models provide state-of-the-art reference points.The frontier comparators are GPT-5.6-Luna, Claude Opus 5, Kimi K3, and GLM-5.3.Source: §3 · p. 11
• The evaluation covers two dense and two mixture-of-experts models, whose initial HarmBench ASRs range from 0.25% for Safin to 53.50% for Ministral.Qwen3.6-27B and Safin-1 begin with ASRs of 3.75% and 0.25%, while Ministral-3-14B-Reasoning-2512 and DeepSeek-V4-Flash-0731 begin at 53.50% and 33.25%.Source: §3 · p. 11
• ReSI uses DARWIN, JailbreakSkill, and MAGIC for red-teaming, with SInternal-style critic SFT, A3-style response SFT, and AlphaAlign-based safety GRPO in its alignment pool.GPT-4o guides recipe proposals and refinement; the loop stops when no candidate passes the gate, after one to three accepted rounds across the four models.Source: §3 · p. 11

4.1. Resistance: in-distribution safety improvements and retention of existing defenses

• Across four models, ReSI improved WildJailbreak safety and retained or strengthened HarmBench defenses, outperforming A3 and nearly always MAGIC.Qwen’s WildJailbreak and HarmBench ASRs fell from 11.05% and 3.75% to 0.45% and 0%; Safin retained its 0.25% HarmBench ASR.Source: §4.1 · pp. 11–12

4.2. Resilience: generalization to unseen attack mechanisms

• H-CoT ASRs fell from 72–100% to 0–18%, and mean X-Teaming ASR declined from 86.01% to 31.45% (54.56 points).Source: §4.2 · p. 12
• ReSI-aligned models reached 16.98–43.40% X-Teaming ASR, outperforming every evaluated frontier reference.Source: §4.2 · p. 12

4.3. Retention of general capabilities and benign compliance

• General capabilities and benign compliance were broadly retained, with variation across models and tasks.XS-Test benign-compliance rates rose for Qwen and DeepSeek; IFEval loose instruction accuracy decreased by 1.31–2.07 percentage points across all four models, and Ministral had larger reasoning and knowledge losses.Source: §4.3 · p. 12

Automated red teaming

• Automated red-teaming methods use adversarial generation, target-model feedback, novelty or diversity search, and accumulated attack experience to expose vulnerabilities [48] [49].Examples include GCG [12], PAIR and TAP, Rainbow Teaming, and adaptive multi-turn X-Teaming [23].Source: §5 · pp. 12–13

Safety alignment and iterative adversarial training

• Safety-alignment approaches vary in supervision and objectives, while iterative adversarial methods update defenses using newly generated attacks [57] [58] [59].Unlike methods that iterate within a fixed procedure, ReSI reselects attack sources, alignment methods, data mixtures, and configurations each round.Source: §5 · pp. 13–14

Automated research

• Automated-research systems use experimental feedback to select and refine candidate implementations or research outputs [72] [73] [8].Automated alignment researchers apply this process to safety methods targeting characterized failures.Source: §5 · p. 14

Recursive self-improvement

• Prior recursive self-improvement systems mainly target coding and research performance, whereas ReSI applies a bounded recursive loop to safety [3] [4] [5] [6].ReSI validates each update with a fixed Pareto gate.Source: §5 · p. 14

Discussion and conclusion

• ReSI identifies target-model vulnerabilities with diverse red-teaming methods, develops training recipes through automated research, and validates updates before using them as the next target.The recipe search spans attack data, alignment methods, data mixtures, and training configurations.Source: §6 · p. 15
• Across four models, ReSI strengthens resistance to identified attacks, generalizes gains to unseen strategies, and largely preserves general capabilities and benign-response rates.Source: §6 · p. 15
• Experimental feedback supports selecting and refining alignment methods while assessing safety alongside instruction following and over-refusal.As new vulnerabilities become available, the process can expand safety coverage while checking for regressions; all four loops stopped when no next-round candidate passed the Pareto gate.Source: §6 · p. 15
• Experiments cover a limited set of models and configurations over finite rounds, and ReSI keeps its refinement skills fixed across rounds.Longer runs could examine persistence of gains; future work could update refinement skills from prior-round outcomes.Source: §6 · p. 15

A.1. Attack collection

• Attack sources generate over the full training partition; only attacks receiving safety-judge score 5 are retained, and exact prompt duplicates are removed before pilot sampling.Held-out and evaluation prompts are excluded; pilot counts refer to unique successful records before benign pairing or method-specific supervision.Source: §A.1 · p. 21

A.2. Pilot screening settings

• Table 3 identifies configurable pilot-screening defaults, with attack counts measured as verified successful records per source before method-specific data construction.Table 4 summarizes data and operations for refinement skills; its numerical choices are configurable starting values.Source: Table 3 · p. 21, Table 4 · p. 22
• Each eligible source is paired with every supported alignment method, and the first-round pilots add a shared base of successful attacks from the original safety data.Scaling probes keep the source, method, initial hyperparameters, and parent checkpoint fixed; candidates are ranked by safety gain, with up to K recipes shortlisted.Source: §A.2 · pp. 21–22

A.3. Full-scale refinement settings

• Full-scale refinement uses configurable SFT and GRPO budgets, with S4 continuing safety GRPO from an SFT checkpoint when eligible.S3 uses capped general-task data; S4 screens prompts and retains the SFT checkpoint as a training start, not as an automatically accepted final candidate.Source: §A.3 · pp. 22–23
• Each branch allows up to 3 revisions per training phase, and the search ends when no eligible update remains or after three outer rounds.Incomplete trials are excluded from selection; unhandled errors stop execution without substitute scores.Source: §A.3 · p. 23

B. Experimental Details

• The experimental-details appendix covers data preparation and evaluation protocols, while model-specific configurations are described in Appendix C.The recovery thresholds in Table 5 permit further training, not final acceptance.Source: Table 5 · p. 23, B. Experimental Details · p. 24

• The WildJailbreak training split supplies the safety pool, and 128 held-out requests are reserved for validation and excluded from training-data construction.Source: §B.1 · p. 24
• Final WildJailbreak evaluation uses the separate testing split rather than the training split.Source: §B.1 · p. 24
• Table 6 lists the final evaluation panels, with counts referring to evaluated requests, behaviours, or prompts.Source: Table 6 · p. 24

B.2. Validation and evaluation protocols

• A3 covers all four target models, while MAGIC covers Qwen, Ministral, and DeepSeek but excludes Safin because its backend lacks required RL support.Both baselines share ReSI’s evaluation protocol and target-model generation settings.Source: §B.2 · p. 24

B.2.1. Validation subsets and retention thresholds

• Full-scale selection enforces benign-compliance and instruction-following retention in addition to the safety-screening conditions.Table 7 specifies validation panels and thresholds; constraints apply separately rather than being combined into an aggregate gate.Source: §B.2.1 · p. 24, Table 7 · p. 24
• H-CoT and XS-Test/IFEval validation subsets overlap with final evaluation and informed selection, so these results are not independent held-out evidence.Single-run comparisons do not establish statistical significance.Source: §B.2.1 · p. 25

B.2.2. Benchmark evaluation protocols

• WildJailbreak, HarmBench, and H-CoT responses are scored by GPT-4o at temperature 0, with only score 5 counted as a successful attack.The five-point scale is adapted from X-Teaming.Source: §B.2.2 · p. 25
• X-Teaming evaluation bounds attacks to 10 plans, 7 dialogue turns, and 4 TextGrad iterations, and records the highest response score across the sequence.Source: §B.2.2 · p. 25
• XS-Test classifies benign responses as full compliance, full refusal, or partial refusal; both refusal labels count as over-refusal.The classification measures refusal, not factual correctness.Source: §B.2.2 · p. 25
• GPQA-Diamond and MMLU-Pro report the fraction of extracted answers matching references, while IFEval reports prompt- and instruction-level loose accuracies.The IFEval gate uses loose instruction accuracy.Source: §B.2.2 · p. 25

B.2.3. Generation settings

• The output limit is 32,768 tokens, increased to 65,536 after truncation; Table 8 summarizes evaluation sampling settings.Source: §B.2.3 · p. 26, Table 8 · p. 26

C. Model-Specific Training Recipes

• Training-data counts use prompt intent categories, include replay examples in totals, and define original harmful prompts as successful attacks found directly in the WildJailbreak training split.Harmful includes both unsafe-response and safe-refusal critic examples.Source: C. Model-Specific Training Recipes · p. 27
• The Qwen, Ministral, and selected DeepSeek and Safin recipe tables organize training recipes by outer round or checkpoint selection, with GRPO starting from the preceding SFT model where specified.The Qwen table specifies that GRPO starts from the preceding SFT model; the Ministral table distinguishes round-0 from round-1 GRPO starts.Source: Table 9 · p. 27, Table 10 · p. 27, Table 11 · p. 28
• Shared settings use LoRA rank/scaling 16/32 except Safin at 32/64, with effective batch sizes and optimizer steps for SFT or rollout-training iterations for GRPO.Safety GRPO samples eight responses per prompt at temperature 1.0 and top-p 0.95; harmful-response rewards depend on safety scores, while benign responses also require full compliance.Source: C. Model-Specific Training Recipes · p. 28

1. Propose and compare pilot recipes

• Pilot screening advances JailbreakSkill with critic SFT: both A3 and JailbreakSkill reach zero WJB ASR, while JailbreakSkill has lower H-CoT ASR.The pilot candidates share a common budget, and instruction-following retention remains unresolved at this stage.Source: D. Recipe Search Case Study · p. 29, Table 12 · p. 29

2. Expand and evaluate the shortlisted recipe

• The expanded SFT trial raises H-CoT ASR from 16% to 26% and WJB ASR from 0% to approximately 2.34%; its 92% IF score is six points below the backbone reference.Because data composition and update count also change, the comparison does not isolate scale; the checkpoint is ineligible.Source: D. Recipe Search Case Study · p. 29, Table 13 · p. 29

3. Refine the recipe through GRPO recovery

• The final training path retains the JailbreakSkill source and adds safety GRPO after critic SFT, while excluding the failed SFT checkpoint from final selection.The case traces the recipe from its initial proposal through screening, full-scale training, and recovery.Source: D. Recipe Search Case Study · p. 29

4. Select the completed update

• 0% WJB and 4% H-CoT ASRs at GRPO step 20 satisfy retention constraints and are lower than step 50’s 0.78% and 8%, so step 20 is selected as the next model.Table 13’s retention checks use 100 benign XS-Test requests and 50 IFEval prompts, each allowed to decline by at most three percentage points.Source: D. Recipe Search Case Study · p. 29, Table 13 · p. 29