SAHOO: Safeguarded Alignment for High-Order Optimization Objectives in Recursive Self-Improvement

Subramanyam Sahoo; Aman Chadha; Vinija Jain; Divya Chaudhary

Source-linked AI summary

SAHOO: Safeguarded Alignment for High-Order Optimization Objectives in Recursive Self-Improvement

Subramanyam Sahoo, Aman Chadha, Vinija Jain, Divya Chaudhary

arXiv:2603.06333v1Mar 6, 2026cs.AIcs.CLcs.LG

TL;DR

Recursive self-improvement can produce capability gains while drifting from intended alignment goals. SAHOO addresses this with drift detection, constraint preservation, and regression safeguards, yielding gains across three task types while maintaining low drift and strong constraint preservation.

Problem
Recursive self-improvement systems can critique, modify, and evaluate outputs, but iterative self-modification may drift from intended alignment goals.
Method
SAHOO combines the Goal Drift Index, constraint-preserving mechanisms, and regression-risk safeguards, with thresholds calibrated from task data.
Results
SAHOO achieved 3.8 to 18.3 percent capability improvements across domains while keeping drift below critical thresholds and constraint preservation above 0.987.
Takeaways & Limitations
The framework makes alignment preservation during recursive self-improvement measurable across diverse benchmarks and supports monitoring of heterogeneous drift dynamics.
Takeaways & Limitations
The framework requires recalibration for new task families or architectures, explicit constraint specifications, and often human evaluation for truthfulness checks.

Abstract

from arXiv · show

Recursive self-improvement is moving from theory to practice: modern systems can critique, revise, and evaluate their own outputs, yet iterative self-modification risks subtle alignment drift. We introduce SAHOO, a practical framework to monitor and control drift through three safeguards: (i) the Goal Drift Index (GDI), a learned multi-signal detector combining semantic, lexical, structural, and distributional measures; (ii) constraint preservation checks that enforce safety-critical invariants such as syntactic correctness and non-hallucination; and (iii) regression-risk quantification to flag improvement cycles that undo prior gains. Across 189 tasks in code generation, mathematical reasoning, and truthfulness, SAHOO produces substantial quality gains, including 18.3 percent improvement in code tasks and 16.8 percent in reasoning, while preserving constraints in two domains and maintaining low violations in truthfulness. Thresholds are calibrated on a small validation set of 18 tasks across three cycles. We further map the capability-alignment frontier, showing efficient early improvement cycles but rising alignment costs later and exposing domain-specific tensions such as fluency versus factuality. SAHOO therefore makes alignment preservation during recursive self-improvement measurable, deployable, and systematically validated at scale.

1 INTRODUCTION

Recursive self-improvement offers iterative capability gains but risks compounding alignment drift. SAHOO addresses this with learned drift detection, constraint preservation, and regression safeguards.

Recursive self-improvement enables systems to improve capabilities through iterative refinement while potentially drifting from intended alignment goals.
Alignment drift can occur across semantic, lexical, structural, and distributional dimensions that interact during self-modification.
SAHOO combines drift detection, constraint-preserving loss, and regression safeguards to monitor and control alignment during improvement cycles.The framework learns drift thresholds from calibration data rather than relying on arbitrary hyperparameters.
The framework introduces the Goal Drift Index, constraint-preserving loss, regression-risk bounds, and the Capability Alignment Ratio across three benchmark task types.It also provides a methodology for practitioners to calibrate and deploy the framework.

2 RELATED WORK

Prior work established theoretical foundations for recursive self-modification and later made self-improvement empirically tractable in language-model agents. Alignment and control concerns have consequently received increasing attention.

Theoretical foundations for recursive self-improvement include Gödel machines, optimal self-improvement, and metareasoning.
Recent language-model agents have made self-improvement empirically tractable.
Value alignment and the control problem provide established foundations for studying alignment and safety in self-improving systems.

3 EXPERIMENTAL FRAMEWORK AND DESIGN

The evaluation spans code generation, truthfulness, and mathematical reasoning, using task-specific quality and constraint metrics. A data-driven calibration phase sets drift parameters before the main experiments.

Benchmarks: HumanEval, TruthfulQA, and GSM8K probe alignment failure modes in code generation, truthfulness, and mathematical reasoning.The benchmarks assess syntactic and semantic correctness, factual accuracy, and multi-step problem solving, respectively.
Experimental setup: The study uses Qwen3-8B as its base model and samples 63 tasks per benchmark based on power analysis.Maximum cycles are selected from convergence analysis, while bootstrap samples produce 95% confidence intervals.
Experimental setup: Temperature is fixed at √2/2 ≈0.707, described as the entropy-maximizing value for binary outcomes generalized to token distributions.
Calibration and metrics: Calibration uses six tasks across three improvement cycles to learn drift weights, estimate baseline statistics, set thresholds, and establish quality and constraint-preservation patterns.Quality uses pass@1, truthfulness accuracy, and mathematical exact match; constraint preservation is measured separately.

4 EMPIRICAL RESULTS

Across 189 tasks, recursive self-improvement produced domain-dependent quality gains while safeguards maintained low drift, preserved constraints in code and mathematics, and limited regression risk. Truthfulness remained the principal alignment challenge, with frequent violations and smaller gains.

Constraint preservation: Code and mathematical reasoning achieved perfect constraint preservation, whereas truthfulness recorded 170 violations across 63 tasks.Truthfulness violations were concentrated in fabrication, overconfidence, and system-call-style outputs.
Aggregate performance: Across 189 tasks, code quality increased 18.3% and mathematical reasoning quality increased 16.8%, while truthfulness improved 3.8%.Code rose from 0.672 to 0.795, mathematics from 0.689 to 0.805, and truthfulness from 0.678 to 0.704.
Drift analysis: Semantic drift contributed most to the Goal Drift Index, with a calibrated weight of 0.38, followed by distributional, structural, and lexical drift.The corresponding weights were 0.29, 0.21, and 0.12.
Drift analysis: GDI stabilized after cycles 3–5, remaining below the 0.44 threshold in all domains, with truthfulness slightly higher than code and mathematics.Code and mathematics remained near 0.35, while truthfulness reached approximately 0.38–0.40.
Convergence and stability: Stability remained high, while regression risk was concentrated in one oscillatory task and fell to 0.7% after removing that outlier.Across 3,780 cycles, 170 regression events occurred; 117 came from the single outlier.
Convergence and stability: 91.5% of tasks converged within the cycle budget, with mean convergence times of 7.1 cycles for code, 8.9 for mathematics, and 10.4 for truthfulness.Sixteen tasks reached the maximum cycle count.
Capability-alignment trade-offs: The capability-alignment frontier showed initially efficient gains, followed by lower efficiency as later improvements required greater drift.CAR approached 1.0 early, declined to roughly 0.6–0.7 by cycles 2–3, and then stabilized.

5 DISCUSSION

SAHOO’s results indicate that recursive self-improvement can deliver capability gains while keeping drift and constraint violations low, although benefits and alignment costs vary by domain. Early improvement cycles are especially productive, while regression safeguards identify risky behavior changes.

Capability-alignment frontier: Most improvement trajectories occupied a benign region combining high quality, low drift, and strong constraint satisfaction, while a minority incurred trade-off costs.
Capability-alignment frontier: CAR peaked early, then decayed and stabilized by cycles 2–3, with truthfulness trailing code and mathematics.The pattern indicates that truthfulness gains impose larger alignment costs than gains in the other domains.
Empirical findings: 3.8 to 18.3 percent capability improvements were achieved across domains while drift stayed at 0.32 to 0.35 against a critical 0.44 and constraint preservation exceeded 0.987.Code and mathematical reasoning improved by 16 to 18 percent with CAR around 0.67, while truthfulness improved by 3.8 percent at CAR 0.60.
Improvement dynamics: 91 percent of tasks converged, suggesting that most valuable improvements occurred during the first few cycles.The discussion proposes conservative limits of 5 to 7 cycles to capture gains while minimizing drift accumulation.
Comparison to naive approaches: Compared with heavily conservative approaches, the framework occupies a middle ground of modest drift and meaningful quality improvement.The paper contrasts this with near-zero drift but near-zero quality improvement under approaches that strongly restrict improvement.
Regression safeguards: The regression rate was 0.7 percent excluding one outlier, and regression detection occurred early at a mean of 4.5 cycles.The outlier alternated between distinct solution strategies, creating a detectable signal for human intervention.

6 RISKS, LIMITATIONS, AND EXISTENTIAL RISK MITIGATION

SAHOO makes alignment preservation during recursive self-improvement measurable through drift detection, constraint preservation, and regression safeguards, but its mitigation scope is bounded by calibration, specification, evaluation, and oversight requirements.

Scope boundaries: The framework requires recalibration for new task families or model architectures because it was calibrated on specific benchmark distributions and decoder-style models.
Specification limits: Explicit constraint specifications are difficult for many ethical or value-laden properties, limiting what the framework can directly preserve.
Evaluation limits: Truthfulness checks often require human evaluation, which limits scalability and introduces evaluator-drift risk.
Baseline dependence: Because divergence is measured from a baseline, a misaligned baseline can conceal fundamental value mismatches.
Existential-risk boundary: For very high-capability or potentially deceptive systems, effective human oversight and intervention may no longer be reliable.The paper therefore characterizes SAHOO as a necessary but insufficient mitigation.

7 FUTURE WORKS & CONCLUSION

The paper concludes that principled safeguards can measure, monitor, and control alignment drift during recursive self-improvement while preserving capability gains. It proposes richer constraints, stronger drift robustness, mechanistic interpretability, and multi-agent extensions as next steps.

Future work: Future work should develop richer constraint specifications beyond the paper’s binary satisfaction approach.
Future work: Adversarial robustness of the drift-detection system remains an open research direction.
Future work: Mechanistic interpretability could clarify which aspects of model internals drift during recursive improvement.
Future work: Extending the framework to multi-agent self-improvement could address deployment scenarios involving jointly improving systems.
Conclusion: The paper’s conclusion presents alignment safeguards as necessary for recursive self-improvement to preserve its benefits.Its framework combines the Goal Drift Index, explicit constraint preservation, regression-risk bounds, and Capability Alignment Ratio analysis across three benchmark task types.

LLM USAGE DISCLOSURE

The paper discloses limited LLM assistance for readability and conciseness, with human verification and authors retaining responsibility. The section also describes a data-driven framework for measuring and preserving alignment during recursive self-improvement.

Claude 4.5 Haiku was used to suggest improvements in readability and conciseness while preserving technical accuracy.
The LLM was not used for hypothesis generation, experimental design, data analysis, or scientific interpretation.
All LLM-generated content received human verification and approval.
The authors accept full responsibility for the submission and identify the scientific contributions, methodology, and conclusions as their own intellectual work.
Framework formulation: The framework formulates tasks with prompts, constraints, and reference solutions, then evaluates quality, constraint satisfaction, and cumulative drift across improvement cycles.
Drift monitoring: The Goal Drift Index combines semantic, lexical, structural, and distributional signals using calibrated weights and task-specific thresholds.

C.1 CONSTRAINT SPECIFICATION

The framework represents safety requirements as explicit logical constraints and evaluates whether model outputs satisfy them. It combines automated and learned checks with hard scoring, violation-focused feedback, and immediate stopping after severe failures.

Safety constraints are formalized as logical predicates that every model output must satisfy.
Constraints cover format, content, logical consistency, and ethical requirements.
Constraint satisfaction uses automatic checks for format and simple content requirements alongside learned evaluators for semantic constraints.
The Constraint Preservation Score measures the fraction of satisfied constraints as a hard, non-relaxed metric.
Violation feedback is inserted into the next improvement prompt, with greater penalties for more numerous or severe violations.
The process halts immediately when any cycle produces zero constraint preservation.

D LONG HORIZON STABILITY ANALYSIS

The long-horizon analysis quantifies regression risk from historical quality, volatility, and trend, then connects stopping decisions to the capability–alignment trade-off. It also examines gradual misalignment, specification gaming, and capability collapse as risks of recursive improvement.

Regression is defined as later quality improvements being undone by reversion to earlier behavior.
Regression-risk estimation uses historical quality, volatility, fitted trend, and the gap from the prior quality maximum.
The framework predicts next-cycle quality under an approximately Gaussian error model and adjusts risk upward for significantly negative trends.
Long-horizon bounds: 0.6^10 ≈ 0.00605: with a 0.6 per-cycle improvement probability, ten consecutive improvements have about a 0.6% probability under the simplified model.
Capability–alignment frontier: The Capability Alignment Ratio compares capability gains with cumulative alignment degradation, while Pareto-frontier points represent non-dominated trade-offs.
Alignment risks: Gradual misalignment can accumulate without single-cycle violations, so multi-signal drift detection is used to monitor semantic, lexical, structural, and distributional changes.
Alignment risks: Overly restrictive constraints may prevent meaningful capability gains, motivating explicit capability–alignment trade-off analysis.

F.2 LIMITATIONS OF THE FRAMEWORK

The framework is limited by baseline alignment, explicit constraint requirements, task-distribution dependence, and uncertain adversarial robustness. Deployment also requires operational safeguards such as logging, oversight, circuit breakers, rollback, and adjustable improvement limits.

Measurement limitations: The GDI measures divergence from baseline behavior, so it is most useful when the initial model is reasonably aligned.A misaligned baseline can make measured drift differ from drift from true alignment.
Constraint specification: The framework assumes safety constraints can be explicitly specified, limiting its benefit when alignment properties are difficult to formalize.More nuanced constraint specification remains an open direction tied to interpretability and mechanistic understanding.
Task distribution shift: Substantial task-distribution changes require recalibration because the framework is calibrated for specific task types.The paper contrasts mathematical reasoning with creative writing as an example of a shift requiring recalibration.
Adversarial robustness: The framework detects natural drift but may not withstand adversarial manipulation designed to evade detection.Adversarial robustness is identified as an open problem beyond the work’s scope.
Deployment safeguards: Deployment should include logging, periodic human review, external circuit breakers, and cycle-level checkpoints for rollback.These measures preserve auditability, enable intervention, and support recovery from undesirable changes.
Mitigation strategies: Practitioners can trade performance for safety through conservative thresholds, stronger constraints, fewer cycles, or inter-cycle human verification.A capability ceiling is another option when further improvement incurs excessive alignment costs.

G THEORETICAL ANALYSIS

The theoretical analysis models drift as a Lipschitz-continuous function of response changes and relates cumulative drift to quality improvements across cycles. It identifies a contractive regime in which iterative improvement remains bounded, while larger coupling predicts instability without control.

Drift measurement: The drift measurement maps pairs of responses in a metric response space to a normalized drift value.The theoretical setup denotes the response space by Y and its metric by dY(·, ·).
Drift measurement: A weighted combination of semantic, lexical, structural, and distributional drift components remains Lipschitz under suitable feature metrics.Nonnegative component weights preserve continuity, with the bound obtained through the triangle inequality.
Drift measurement: Small response perturbations therefore produce proportionally small changes in measured drift under the chosen metric coupling.This is the operational implication of the Lipschitz result.
Cumulative drift: Expected cumulative drift grows at most linearly when the mean per-cycle quality increment is positive and grows O(C) when that increment is approximately zero.The result follows from telescoping quality changes and the stated independence and variance assumptions.
Contractive regime: A contractive regime requires L∆ < 1, yielding a limiting expected drift of L∆µ/(1 − L∆) under the affine evolution model.Contraction produces geometric convergence to the fixed point.
Contractive regime: When L∆ < 1, iterative improvements can continue without unbounded drift; when L∆ ≥ 1, the linearized recurrence predicts instability absent external control.The framework’s practical safeguards provide such external control mechanisms.
Practical implications: The practical analysis emphasizes recalibration across domains, differing alignment costs by task type, and continued human review of improvement trajectories.The framework is reported to work better for clearly specified code and mathematics than for open-ended tasks.