PaperlayerView exact revision on arXiv

Source-linked AI summary

Hyperagents

Jenny Zhang, Bingchen Zhao, Wannan Yang, Jakob Foerster, Jeff Clune, Minqi Jiang, Sam Devlin, Tatiana Shavrina

arXiv:2603.19461v1cs.AI

TL;DR

Existing self-improvement systems rely on fixed meta-level mechanisms and often depend on alignment between task performance and self-modification skill. This paper introduces editable hyperagents that combine task and meta agents, forming DGM-H through open-ended exploration. Across diverse domains, DGM-H improves task performance and its ability to generate improvements, with meta-level gains transferring across domains and accumulating across runs.

  • Problem

    Existing self-improvement systems rely on fixed, handcrafted meta-level mechanisms, and prior approaches may depend on alignment between task performance and self-modification skill.

  • Method

    The paper combines task and meta agents in a single editable hyperagent and extends DGM with open-ended exploration and metacognitive self-modification.

  • Results

    Across diverse domains, DGM-H produces substantial and generalizable gains in task performance and self-improvement ability, with improvements transferring across domains.

  • Takeaways & Limitations

    DGM-H continually improves both its solutions and the process by which it generates future improvements.

  • Takeaways & Limitations

    DGM-H uses a fixed task distribution and fixed outer-loop components such as parent selection and evaluation protocols.

Abstract

from arXiv · show

Self-improving AI systems aim to reduce reliance on human engineering by learning to improve their own learning and problem-solving processes. Existing approaches to self-improvement rely on fixed, handcrafted meta-level mechanisms, fundamentally limiting how fast such systems can improve. The Darwin Gödel Machine (DGM) demonstrates open-ended self-improvement in coding by repeatedly generating and evaluating self-modified variants. Because both evaluation and self-modification are coding tasks, gains in coding ability can translate into gains in self-improvement ability. However, this alignment does not generally hold beyond coding domains. We introduce \textbf{hyperagents}, self-referential agents that integrate a task agent (which solves the target task) and a meta agent (which modifies itself and the task agent) into a single editable program. Crucially, the meta-level modification procedure is itself editable, enabling metacognitive self-modification, improving not only the task-solving behavior, but also the mechanism that generates future improvements. We instantiate this framework by extending DGM to create DGM-Hyperagents (DGM-H), eliminating the assumption of domain-specific alignment between task performance and self-modification skill to potentially support self-accelerating progress on any computable task. Across diverse domains, the DGM-H improves performance over time and outperforms baselines without self-improvement or open-ended exploration, as well as prior self-improving systems. Furthermore, the DGM-H improves the process by which it generates new agents (e.g., persistent memory, performance tracking), and these meta-level improvements transfer across domains and accumulate across runs. DGM-Hyperagents offer a glimpse of open-ended AI systems that do not merely search for better solutions, but continually improve their search for how to improve.

1 Introduction

Existing self-improvement systems use fixed meta-level mechanisms, while the DGM-H introduces hyperagents whose editable self-improvement process can improve across diverse domains.

  • Fixed meta agents constrain how base systems can be improved because their designs define the boundaries of modification.
  • The DGM demonstrates open-ended coding self-improvement, but its handcrafted instruction-generation mechanism remains fixed and bottlenecks further improvement.
  • Hyperagents combine task and meta agents in one self-referential, modifiable program, allowing both task-solving and future-improvement procedures to change.
  • DGM-H retains open-ended exploration while making the meta-level improvement mechanism editable through metacognitive self-modification.
  • Across coding, paper review, and robotics reward design, DGM-H improves task performance and self-improvement ability, with gains transferring to held-out tasks and across domains.

2 Related Work

Prior self-improving systems often retain human-designed learning or modification mechanisms and rely on task-domain alignment; hyperagents make the self-modification process editable instead.

  • Open-ended learning systems continually generate increasingly complex artifacts, but compounding improvement in their discovery processes remains an important next step.
  • Prior self-improvement methods modify weights, learning dynamics, architectures, prompts, or behavior while generally retaining fixed learning algorithms or mechanisms.
  • Self-referential meta-learning studies systems that improve the mechanisms by which learning occurs across neural, evolutionary, and foundation-model-based approaches.
  • Prior work relies on alignment between evaluation-task skills and self-modification skills, whereas hyperagents make the self-modification mechanism fully modifiable across computable tasks.

3 Methods

DGM-H combines task execution and agent generation in editable hyperagents, then uses open-ended archive-based exploration and metacognitive self-modification to evolve them.

  • The system evaluates new hyperagents on computable tasks after generating code and meta-agent modifications.
  • DGM-H preserves DGM’s open-ended exploration by maintaining an archive of generated hyperagents as stepping stones for future gains.
  • A hyperagent unifies task execution and agent generation in a single modifiable program, removing the need for task and self-modification skills to be aligned.
  • Hyperagents combine task and meta agents so the program can modify both task-solving behavior and the procedures generating future modifications.
  • Metacognitive self-modification makes the self-improvement mechanism itself editable alongside the task-performing agent.

4 Experiment Setup

The experiments compare DGM-H with ablations, DGM, and static domain-specific solutions across coding, paper review, robotics reward design, and Olympiad-level math grading.

  • DGM-H is initialized with a simple hyperagent using a frozen foundation model and external tools, leaving a broad search space for improvement.
  • Baselines isolate self-improving meta agents, open-ended exploration, and metacognitive self-modification, while also including the original DGM and static solutions.
  • The evaluation spans coding, paper review, robotics reward design, and Olympiad-level math grading, with staged evaluation used to reduce computational cost.
  • Paper review: Paper review predicts binary acceptance decisions against observed outcomes in a subjective setting without objective ground truth.
  • Olympiad-level math grading: Olympiad-level math grading predicts discrete scores for candidate solutions using problems, references, and grading guidelines, measured against expert grades.

5 Results

Across coding and non-coding domains, DGM-H improves task performance and the ability to generate better agents, with learned meta-level improvements transferring across domains and potentially accumulating across runs.

  • Overall results: Across five-run experiments, DGM-H shows general self-improvement at both task and meta levels, with task gains transferring to held-out tests and meta gains transferring across domains.The authors also report that self-improvements from one run can potentially accelerate subsequent runs.
  • Task performance: On held-out test tasks, DGM-H improves paper review from 0.0 to 0.710 and robotics reward design from 0.060 to 0.372, surpassing their cited static or default baselines.The corresponding baselines are 0.630 for paper review and 0.348 for robotics reward design.
  • Qualitative analysis: DGM-H task agents move beyond surface-level prompt changes toward reusable decision machinery, including structured review pipelines and robotics knowledge bases.These changes are described as producing more consistent judgments, eliminating compilation failures, and reducing reward misspecification.
  • Task performance: DGM-H significantly outperforms the original DGM in paper review and robotics reward design, where the original DGM’s coding-specific mechanism performs poorly without domain customization.The original DGM achieves 0.0 on both domains, whereas DGM-custom performs substantially better after manual customization.
  • Improving the ability to improve: Holding the initial meta agent fixed, DGM-H w/o self-improve generates substantially better agents from transfer hyperagents, reaching imp@50 0.630 on unseen Olympiad-level math grading.The corresponding initial and transfer-agent baselines achieve little or no improvement, while the transfer comparison is statistically significant (p < 0.05).

6 Safety Discussion

DGM-H can autonomously modify its behavior and improvement mechanisms, creating safety challenges as capabilities grow. The experiments therefore use strict sandboxing and controlled evaluations, but these safeguards may become harder to sustain.

  • DGM-H experiments execute agent-generated code in sandboxed environments with resource limits, including timeouts and restricted internet access.These controls are intended to prevent unintended side effects, contain failures, and ensure controlled evaluations.
  • Self-modifying systems may evolve faster than humans can audit or interpret, complicating oversight as capabilities increase.
  • As self-improving systems grow more capable, current safety safeguards may become increasingly strained or infeasible.The paper highlights trust, oversight, transparency, and societal deliberation as continuing concerns.

7 Limitations and Conclusion

DGM-H combines open-ended evolution with metacognitive self-modification, producing transferable and compounding gains across domains. The paper concludes that this framework is promising but remains bounded by fixed task distributions and outer-loop components.

  • 7 Limitations and Conclusion: DGM-H evolves an archive of self-improving hyperagents for any computable task, improving both task performance and its improvement mechanism.
  • 7 Limitations and Conclusion: Meta-level gains transfer across domains and compound across runs alongside substantial and generalizable task-performance improvements.
  • 7 Limitations and Conclusion: The current system keeps the task distribution, parent selection, and evaluation protocols fixed, limiting full self-modifiability and truly unbounded progress.The paper proposes co-evolving tasks and enabling modification of outer-loop search and evaluation components as future directions.
  • 7 Limitations and Conclusion: With appropriate safety considerations, DGM-H suggests a path toward systems that continually improve their ability to self-improve.
  • Appendix Details: The appendix provides implementation details for the initial hyperagent, parent selection, and DGM-H pseudocode.
  • A.2 Parent Selection: DGM-H uses parent selection that balances performance and exploration, then evaluates valid self-modified children before adding them to the archive.For multi-domain runs, selection uses average performance across domains and shared evaluations can support mechanisms benefiting multiple domains.

B Baseline Details

The baselines isolate the contributions of self-improving agents and open-ended exploration, while DGM variants differ in how task solving and improvement generation are coupled. The original DGM relies on handcrafted instruction generation, whereas DGM-H uses a modifiable hyperagent architecture.

  • Baseline Algorithms: Baseline pseudocode compares DGM-H with versions lacking self-improving meta agents or open-ended exploration, alongside the original DGM.
  • Original DGM: The original DGM uses a handcrafted instruction-generation step before modifying and evaluating each candidate agent.
  • DGM Variants: DGM-H integrates task and meta agents in one editable program, enabling metacognitive self-modification beyond fixed instruction generation.
  • Instruction Generation: The coding-agent instruction generator analyzes implementation code, logs, task information, predicted patches, and test results to propose one general improvement plan.
  • DGM-custom: DGM-custom supplies both coding-agent and task-agent implementations plus task reports and logs to a domain-customized instruction prompt.

C.1 Polyglot

The Polyglot domain evaluates agents that modify software repositories according to natural-language instructions, using predefined tests to assess correctness. Evaluation proceeds from an initial training subset to broader training and unseen benchmark tasks.

  • Task and evaluation: Polyglot tasks require modifying a software repository to implement a natural-language instruction and produce a patch.The modified repository is evaluated with a predefined test suite.
  • Task and evaluation: 60 training tasks are used, beginning with an initial 10-task evaluation and expanding to 50 additional tasks for agents exceeding 40% success initially.Agents that do not pass the initial threshold are not expanded to the remaining training tasks.
  • Task and evaluation: The full Polyglot benchmark contains 165 unseen tasks used for final evaluation of generated agents.
  • Related evaluation setting: The paper-review domain instead presents full AI research papers for binary accept-or-reject prediction against real conference decisions.Its performance measure is classification accuracy relative to labels from ICLR and NeurIPS decisions.

D.3 Improvement@k Metric

The improvement@k metric measures how much a fixed meta agent can improve an initial task agent within a bounded number of generation steps. The section also describes selecting transfer agents by descendant growth, favoring agents that produce strong future improvements rather than merely high current scores.

  • Improvement@k: The metric evaluates agents generated from an initial task agent by a fixed meta agent using a fixed evaluation procedure.The meta agent remains fixed while the generation algorithm produces up to k new task agents.
  • Improvement@k: imp@k is the maximum evaluation improvement obtained by generated task agents within k modification steps relative to the initial agent.Higher imp@k indicates stronger agent-generation capability under the same computational budget and generation procedure.
  • Metric limitation: imp@k treats performance gains as linear, so it may underestimate improvements achieved near performance saturation.The authors state that matched initial conditions and fixed evaluation budgets preserve its usefulness for relative comparisons in this work.
  • Transfer Agent Selection: Transfer-agent selection favors archive nodes whose descendants achieve strong improvements within fewer self-modification steps, rather than nodes with only high current scores.The growth score discounts improvements by descendant distance and uses γ = 0.6 in the experiments; candidates must have at least 3 descendants.

E.2 Qualitative: Improving Task Performance

Across paper review, robotics reward design, and Olympiad-level math grading, DGM-H improves task performance through cumulative changes to task behavior and supporting meta-level machinery.

  • Cross-domain pattern: DGM-H consistently improves task performance across paper review, robotics reward design, and Olympiad-level math grading.Archive trees and progress plots visualize these improvements across domains.
  • Paper review: In paper review, DGM-H replaces unreliable attitude-based prompting with a structured two-stage evaluation process.The transition followed analysis showing that “rigorous and critical” reviewer personas encouraged unreliable strictness.
  • Robotics reward design: In robotics reward design, successive generations improve performance by grounding the agent in increasingly accurate environment documentation and reward-design constraints.Later revisions add tighter constraints, code examples, reward formulations, and scaling ranges.
  • Olympiad-level math grading: In Olympiad-level math grading, DGM-H progressively develops explicit grading categories, decision procedures, calibration, rubrics, and checklists.These changes address the tendency to solve problems instead of grading them and add boundary-case guidance.
  • Cumulative improvement: Best-performing agents depend on insights and infrastructure accumulated across earlier generations rather than isolated single-step edits.Meta-level improvements include evaluation-analysis utilities and persistent memory that enable cumulative recombination.

E.3.1 Better Evaluation Analysis

The meta agent evolves from unguided code modification toward data-driven evaluation analysis, allowing later generations to use performance metrics and failure patterns for targeted improvements.

  • Evaluation analysis: The meta agent autonomously evolves from a simple instruction to modify any code into a mechanism that analyzes previous evaluation results before changing the system.The _analyze_evaluations() method processes stored evaluation data and summarizes successes, failures, and errors.
  • Evaluation analysis: Evaluation summaries quantify success rates and expose failure patterns for subsequent improvement instructions.The evolved prompt incorporates evaluation context so the meta agent can identify weaknesses and opportunities for improvement.
  • Evaluation analysis: The resulting modifications enable later generations to retrieve performance metrics and identified failure patterns, making improvements more targeted.This principle is discovered independently rather than manually specified.

E.3.2 Compute-Aware Strategic Planning

DGM-H develops strategic machinery that adapts improvement decisions to available compute, diagnoses degenerate behavior, tracks progress, and reuses modular prompt infrastructure.

  • Compute-aware strategic planning: DGM-H adapts its improvement strategy to remaining iterations, making ambitious changes early and conservative refinements late.The system balances exploration and exploitation without that trade-off being manually specified.
  • Compute-aware strategic planning: Compute-aware planning expands self-modification beyond a fixed heuristic by varying exploration intensity, variant generation, and consolidation timing.The resulting strategy can emulate optimization procedures such as evolutionary search, hill climbing, or population-based exploration.
  • Automated bias detection and correction: In paper review, automated label-distribution analysis detects classification collapse and supplies bias information for corrective action.One observed case improved from 49% accuracy with a 99% acceptance rate to 63% accuracy after correction.
  • Performance tracking: Performance-tracking infrastructure records metrics across generations, identifies improvement trends, and supports data-driven decisions about future modifications.The tracker stores generation scores and computes best, worst, average, and moving-average improvement statistics.
  • Reusable prompt infrastructure: Reusable prompt components abstract common reasoning, few-shot, JSON-output, and domain-specific patterns into configurable templates.The implementation applies modular software-engineering patterns including DRY and the factory pattern.

E.3.7 Persistent Memory

DGM-H develops persistent external memory that preserves structured insights across generations, enabling temporal reasoning, cross-generation learning, and avoidance of repeated mistakes.

  • Persistent memory: The hyperagent autonomously creates persistent memory mechanisms for storing and retrieving insights across sessions and generations.The memory tool stores timestamped entries and supports retrieval by key.
  • Cross-generation learning: Memory supports cross-generation learning by letting later generations use earlier successes and failures to guide subsequent improvements.Generation 67 is informed by generation 55’s success and generation 66’s failure.
  • Persistent knowledge: Timestamps provide temporal context, while a shared knowledge base preserves lessons across code changes and helps prevent repeated mistakes.These functions allow the system to reason about recency and retain key learnings as the code evolves.
  • Structured knowledge: Stored knowledge includes synthesized performance insights, causal hypotheses, and strategic plans rather than raw evaluation numbers alone.Examples contrast Gen55’s accuracy with its harshness and propose combining it with Gen64’s balance.
  • Persistent memory: External memory extends the hyperagent’s cognitive capabilities beyond a single inference session.The paper presents this as a consequence of creating external memory storage.

E.4 Olympiad-level Math Graders

BetterGrader improves Olympiad-level math grading by recognizing nuanced Almost and Partial solutions more accurately while preserving strong Correct and Incorrect judgments.

  • The confusion matrices show fewer collapses of intermediate solutions into extreme Correct or Incorrect labels, while strong performance on both extreme labels is maintained.
  • BetterGrader increases true positives for Almost (43 vs. 13) and Partial (72 vs. 47) compared with ProofAutoGrader.These gains correspond to recall improvements of +18.52% for Almost and +10.50% for Partial.
  • BetterGrader improves Almost and Partial F1 by +0.203 and +0.109, respectively.
  • BetterGrader raises overall accuracy by +4.06% despite a modest Correct-recall decrease of −6.00%.ProofAutoGrader’s Correct recall was already 91.67%.

E.5 Modifying Parent Selection

DGM-H can modify parent selection to improve over random exploration, but its automatically evolved mechanism remains below a carefully handcrafted strategy.

  • Experimental setup: DGM-H starts from random parent selection and allows its latest agent to modify the mechanism across repeated runs.The comparison includes random selection throughout, and each method is repeated across 5 runs.
  • Performance comparison: 0.491 was the DGM-H’s average test performance with modifiable parent selection, versus 0.428 with random selection.The reported confidence intervals were 0.387–0.512 and 0.407–0.432, respectively; differences were not statistically significant (p > 0.05).
  • Performance comparison: The handcrafted score-child-prop mechanism outperformed automatically modified parent selection on paper review and robotics reward-design test tasks.
  • Evolved mechanisms: Across runs, DGM-H replaces random sampling with UCB-style exploration bonuses and temperature-controlled softmax selection.These strategies combine performance information with explicit exploration or probabilistic trade-offs between exploration and exploitation.
  • Evolved mechanisms: Later mechanisms incorporate adaptive temperature, diversity, recency, elite performance, and stagnation-sensitive exploration signals.
  • Limitation: The automatically modified strategy does not consistently identify a strong parent-selection policy across runs, despite recreating and extending classic selection components.

F Additional Safety Discussion

The DGM-H optimizes fixed benchmarks and evaluation signals, so its behavior can reflect human biases and exploit weaknesses in the metrics used to guide improvement.

  • Benchmark dependence: Because task definitions and evaluation criteria remain fixed, DGM-H optimizes existing objectives rather than changing the underlying tasks.
  • Bias reflection: In paper review, DGM-H learns acceptance decisions reflecting human review data, potentially clarifying and amplifying its embedded norms and biases.
  • Bias reflection: If benchmarks encode undesirable biases or misaligned incentives, DGM-H may faithfully optimize for and exacerbate them.
  • Evaluation gaming: Optimizing empirical evaluation signals can produce higher measured performance while deviating from the benchmark’s intended objective.
  • Evaluation gaming: Mitigation requires robust, diverse, refreshed evaluations, held-out tests, complementary metrics, and human oversight.
Loading 2603.19461v1…