PaperlayerView exact revision on arXiv

Source-linked AI summary

AI & Human Co-Improvement for Safer Co-Superintelligence

Jason Weston, Jakob Foerster

arXiv:2512.05356v2cs.AI

TL;DR

The authors argue that unguided self-improvement poses risks and is not the fastest or safest route to superintelligence. They propose human-AI collaboration on AI research, expecting it to accelerate progress while offering greater steerability and focus on human-centered safety.

Abstract

from arXiv · show

Self-improvement is a goal currently exciting the field of AI, but is fraught with danger, and may take time to fully achieve. We advocate that a more achievable and better goal for humanity is to maximize co-improvement: collaboration between human researchers and AIs to achieve co-superintelligence. That is, specifically targeting improving AI systems' ability to work with human researchers to conduct AI research together, from ideation to experimentation, in order to both accelerate AI research and to generally endow both AIs and humans with safer superintelligence through their symbiosis. Focusing on including human research improvement in the loop will both get us there faster, and more safely.

The quest for Self-Improving AI

AI self-improvement has broadened from optimizing model weights to learning across more aspects of a system, while autonomous architecture and code improvement remains nascent. Because unguided AI autonomy is described as dangerous, the authors propose human-AI collaboration on AI research.

The quest for Self-Improving AI

  • Self-improvement has expanded from optimizing model weights to learning across data, methods, objectives, update rules, and implementation.The passage describes scaling to larger models before the current shift toward improving more aspects through learning.Source: The quest for Self-Improving AI · p. 1
  • Models now create training data, challenge themselves, and evaluate or reward their own performance; some such approaches are standard building blocks for frontier models.The passage says these axes have already produced sizable performance gains.Source: The quest for Self-Improving AI · p. 1
  • Autonomous improvement of AI architectures and code remains in its infancy, while unguided AI autonomy is described as dangerous for humankind.The authors propose focusing on humans and AI working together to solve AI problems while there is still time before AI eclipses humans in all endeavors.Source: The quest for Self-Improving AI · p. 1

A better quest for humans: Co-Improving AI

The authors advocate co-improvement: human researchers and AI collaborate on AI research, with humans steering its direction. They envision this collaboration advancing AI research and supporting safer, human-centered benefits.

A better quest for humans: Co-Improving AI

  • Co-improvement is bidirectional progress: humans guide AI improvement, while AI augments human cognition, research, and capabilities, unlike autonomous self-improvement.The authors frame human participation as a way to steer research toward positive outcomes for humanity.Source: A better quest for humans:… · pp. 1–2
  • Co-improvement is expected to speed the search for paradigm shifts and offer more transparency, steerability, and focus on human-centered safe AI than direct self-improvement.The authors note that paradigm-shifting advances have required substantial researcher effort, including intermediate results and dead ends.Source: A better quest for humans:… · p. 2
  • The proposed focus is measuring and improving AI research-collaboration skills across research activities, prioritizing research quality over automated artifact production.Activities include identifying problems, creating data and benchmarks, innovating methods, designing and running experiments, and evaluating results to refine the process.Source: A better quest for humans:… · pp. 2, 4
  • The first goal is improving AI research; the envisioned endpoint is superintelligence developed with humans helping guide each step and consider safety and societal harms.The authors also envision increasing humanity's collective knowledge at each step.Source: A better quest for humans:… · p. 4
  • The authors envision extending co-improvement from AI research to other research and topics important to humanity, with AI helping extend human abilities and knowledge.They call AI helping humanity achieve abilities beyond current ones co-superintelligence.Source: A better quest for humans:… · p. 4
  • Although greater capabilities can increase potential harms, collaboration could help find and implement capabilities, safety procedures, values, constraints, and constitutions for safer models.The passage presents this as an optimistic possibility if done correctly.Source: A better quest for humans:… · p. 4
  • For broader societal problems, human-AI collaboration could aggregate viewpoints, structure debate, and help people reach actionable conclusions.The passage contrasts this collaborative approach with an AI overlord dictating best practices.Source: A better quest for humans:… · p. 4
  • The authors support managed openness when needed to combat societal harms, with the question remaining an ongoing discussion as capabilities increase.They connect openness in science with reproducibility, verification, and advancing collective knowledge.Source: A better quest for humans:… · p. 4

Relation to other existing positions

The position relates co-improvement to human-centered and cooperative AI, while focusing specifically on collaborative research toward (co-)superintelligence. It contrasts autonomous self-improvement with a vision in which humans remain necessary, maximally augmented participants in decision-making.

Relation to other existing positions

  • Unlike human-centered AI positions [55], this position specifically targets (co-)superintelligence through collaborative research.It also relates to cooperative AI, scalable oversight [61], and work on finding flaws in AI designs.Source: Relation to other existing positions · pp. 4–5
  • The AI co-scientist [59] is a related practical system, but it targets biomedical research rather than AI research.Cooperative AI addresses finding common ground with machines more broadly.Source: Relation to other existing positions · p. 4
  • The authors argue that research collaboration should tackle misalignment and begin before systems are fully deployed.They connect bidirectional collaboration with finding flaws in AI designs.Source: Relation to other existing positions · p. 5
  • Autonomous self-improvement appears in proposed approaches [63], including experience-driven learning and visions in which humans play little role.The authors note that experience-driven autonomy offers fewer opportunities for human intervention.Source: Relation to other existing positions · p. 5
  • The authors envision humans remaining necessary, maximally augmented participants in economic, scientific, and other decision-making.They call on the AI community to embrace and implement this vision.Source: Relation to other existing positions · p. 5

Conclusion

The authors argue that autonomous self-improving AI is neither the fastest nor the safest route to superintelligence. They advocate collaborative AI research with humans, which could help create highly capable systems that address important societal problems.

Conclusion

  • The authors argue that autonomous self-improving AI is neither the fastest nor the safest way to achieve superintelligence.They describe that existing goal as misguided.Source: Conclusion · p. 5
  • They advocate building AI that collaborates with human researchers to develop more collaborative, capable, and safer AI.The proposed collaboration centers on conducting research together.Source: Conclusion · p. 5
  • The authors say this goal could unlock highly capable AIs that collaborate with people on important goals and societal problems.This is presented as a possible future outcome.Source: Conclusion · p. 5

Related papers

Loading 2512.05356v2…