Source-linked AI summary
From AGI to ASI
Tim Genewein, Matija Franklin, Alexander Lerchner, Laurent Orseau, Samuel Albanie, Adam Bales, Cole Wyeth, Stephanie Chan, Iason Gabriel, Joel Z. Leibo, Allan Dafoe, Marcus Hutter, Thore Graepel, Shane Legg
TL;DR
The report asks how AI may progress beyond human-level AGI and what could slow that transition. It maps four pathways and concludes that continued progress into ASI within the next decade or two cannot easily be dismissed.
Problem
The report examines how AI may progress beyond human-level AGI and which frictions could slow or halt that progress.
Method
The report characterizes intelligence using a qualitative continuum grounded in the Legg-Hutter framework and maps four pathways from AGI to ASI.
Results
Large enough groups of human-level AGI would likely produce generally superhuman capabilities, while recursive self-improvement could make the transition to ASI rapid.
Takeaways & Limitations
Preparing for a post-AGI world requires diverse forecasts and scenarios, continual benchmarking, and monitoring as evidence changes.
Takeaways & Limitations
Current models may be approaching human-defined knowledge boundaries rather than rapidly developing grounded discovery of novel concepts and causal structure.
Abstract
from arXiv · showhide
Over the last decade, building human-level artificial general intelligence has moved from far-fetched speculation to being a concrete next-decade target for many of the largest AI organisations. Achieving this goal would have profound and far-reaching impacts on human society, which raises many complex questions for the decade ahead. This report investigates how AI itself might continue to develop in a post-AGI world along the continuum of machine intelligence. The endpoint of this continuum, Universal AI, is theoretically well understood, which provides some formal grounding for the main focus of this report: the transition from human-level AGI to artificial general superintelligence, which, intuitively, can be understood as a system that is more intelligent and cognitively capable than large organisations of humans. After characterizing ASI, the report discusses four potential pathways from AGI to ASI: scaling AGI, AI paradigm shifts, recursive improvement, and ASI emerging from large-scale multi-agent collectives. The report then discusses possible frictions and bottlenecks along these pathways. Determining whether the impact of these frictions will be negligible or substantial raises a number of concrete open research questions. Due to large uncertainties for predicting ASI progress, it cannot be ruled out that AI progress might continue to accelerate over the next years. This could imply that the image of a single transformative step change, caused by the introduction of human-level AGI into our society, could be inaccurate. More apt might be the prospect of a series of transformative societal changes caused by AI-enabled progress and breakthroughs across many areas of science and technology. Preparing for this prospect requires a massively interdisciplinary endeavour of global scope and interest.
1. Summary Instructions
The report frames the transition from human-level AGI to ASI, examining four potentially parallel pathways and the frictions that could constrain them. It argues that uncertain AI progress may produce successive transformative changes rather than one discrete societal step change.
- Framing AGI and ASI: The report informally characterizes AGI and ASI, while emphasizing advantages of digital intelligence that increase with more compute.ASI is understood as intelligence exceeding the cognitive capabilities of large human organizations.
- Pathways from AGI to ASI: Four pathways from AGI to ASI are examined: scaling AGI, AI paradigm shifts, recursive improvement, and large-scale multi-agent collectives.The pathways are not mutually exclusive and may occur in parallel.
- Frictions and bottlenecks: The report analyzes potential frictions and bottlenecks along these pathways, whose eventual significance remains uncertain.Whether these constraints are negligible or substantial motivates concrete open research questions.
- Uncertain progress: Because ASI progress is difficult to predict, AI development could continue accelerating over the next years.The report therefore does not rule out increasingly rapid progress after AGI.
2. Introduction: Life as we don’t know it?
This section frames the report around AI progress beyond human-level AGI, mapping possible pathways and frictions while emphasizing major uncertainty about whether progress will plateau, accelerate, or become recursively self-reinforcing. It highlights rapidly growing compute and investment, but notes that translating effective compute into new capabilities and societal change remains unresolved.
- Navigating uncertainty: The report maps technological pathways for AI progress beyond human-level AGI and identifies frictions that could slow or halt them, yielding concrete open research questions.The analysis is intended to remain independent of when humanity reaches AGI.
- Rates of progress: 10× per year is a conservative estimate for combined growth from the three compounding factors, although uncertainty could make the overall rate significantly larger or smaller.The estimate follows 1.5 ∗ 2.5 ∗ 3 = 11.25, rounded down.
- Is the Singularity near?: It remains unclear whether effective-compute growth will produce slow capability gains through diminishing returns or exponential gains proportional to effective compute.Even without novel capabilities in individual frontier models, more, faster, or longer-running instances could still increase overall capabilities and applications.
- Is the Singularity near?: AI systems that accelerate AI research could create recursive improvement loops and super-exponential dynamics, but sustained hyperbolic growth is a strong assumption.Such feedback underlies scenarios of rapid AI takeoff or intelligence explosions, while the duration and strength of the effect remain uncertain.
- Navigating uncertainty: AI development might continue without major blockers through at least the end of this decade, potentially implying 10,000-fold growth in effective compute relative to today.Predicting AI-driven scientific and technological acceleration requires weighing AI’s contribution against rising research effort and economic inputs.
3. Characterizing Artificial Superintelligence
The section defines AGI as roughly human-level general intelligence and ASI as broadly superhuman intelligence across virtually all human-relevant tasks, using the Legg-Hutter score to provide formal grounding. It distinguishes ASI from Universal AI, while noting that human-relative thresholds and concrete capability profiles create important limitations.
- Formal grounding: The Legg-Hutter score formalizes intelligence as average performance across all computable tasks, providing qualitative grounding for coarse AGI and ASI characterizations.Simpler tasks receive greater weight in the average because weighting is based on lower Kolmogorov complexity.
- Definitions: AGI denotes roughly median human-level intelligence on most cognitive tasks, whereas ASI denotes superhuman abilities across virtually all human-interest domains.The first AGI may already be superhuman on many tasks, but narrow-domain systems such as AlphaFold and AlphaGo do not qualify as ASI.
- Definitions: ASI is significantly more capable than human-level AGI across the board and may comprise millions of interacting instances operating in parallel.This framing excludes systems that are superhuman only in single domains and makes cooperative design and evaluation an important practical problem.
- Universal AI: Universal AI is the theoretical endpoint of Legg-Hutter intelligence, superior to ASI in data efficiency and general capabilities but incomputable and approximable only from below.It is formally defined through the AIXI agent as an agent that maximizes the Legg-Hutter score.
- Limitations: The Legg-Hutter framework is not intended literally, because restricting tasks to current and future human interest could alter its formal guarantees.Specialized algorithms may outperform AIXI on particular benchmarks, while broader task sets make the universal comparison more relevant.
- Limitations: Human-relative AGI and ASI thresholds are moving targets, while concrete systems may exhibit jagged, non-uniform capability profiles despite smooth idealized intelligence measures.Human capability can rise through technology, education, and cultural artifacts, complicating fixed comparisons between humans and machines.
4. Universal AI — An Informal Overview
Universal AI, formalized by the AIXI framework, provides the strongest known theoretical upper bound and a formal basis for understanding machine intelligence. However, its incomputability and difficulty of practical implementation leave substantial uncertainty about how, or whether, current AI paradigms can approach ASI.
- Universal AI — An Informal Overview: Universal AI is mathematically well understood, but deriving practical algorithms that scale remains elusive, despite progress through more realistic computable variants.The gap between AIXI theory and today’s AI practice remains substantial.
- Universal AI — An Informal Overview: AIXI models a general agent that interacts sequentially with an unknown environment, updates Bayesian beliefs over computable environments and rewards, and plans using those beliefs.Its priors follow Solomonoff’s Universal Prior, assigning higher probability to lower-complexity hypotheses.
- Universal AI — An Informal Overview: AIXI’s optimality guarantee underpins the Legg-Hutter score, a formal and quantitative measure of machine intelligence over computable environments and tasks.The framework generalizes across a broader class of environments than standard machine-learning and reinforcement-learning assumptions.
- Universal AI — An Informal Overview: AIXI and its associated intelligence measure are incomputable, although approximating algorithms can improve with additional compute and runtime.These algorithms remain impractical, and brute-force implementations would require rapidly growing computational resources.
- Universal AI — An Informal Overview: Universal AI and amortized inference provide theoretical arguments that current AI paradigms could potentially reach ASI without fundamental theoretical blockers.The arguments are incomplete and inconclusive, and fundamental shortcomings of today’s paradigm may still emerge.
- Universal AI — An Informal Overview: AIXI’s limitations have motivated alternative frameworks, including reflective oracles, logical induction, Gödel machines, and computational mechanics.These approaches provide complementary perspectives on reflection, self-reference, and causal structure.
5. Technological Pathways and Potential Bottlenecks to ASI
The section examines four potentially parallel pathways from AGI to ASI: continued scaling, paradigm shifts, recursive improvement, and large-scale multi-agent collectives. It highlights major uncertainties and bottlenecks, including data limits, physical constraints, and poorly understood improvement dynamics.
- Overview: Four potentially parallel pathways connect AGI to ASI: scaling compute, data, and models; paradigm shifts; recursive self-improvement; and multi-agent collectives.Scaling is the only pathway that currently supports forecasting models fitted to historical data.
- Scaling AGI: Scaling follows approximate power laws, but it remains unresolved whether quantitative gains in open-ended search and self-improvement can reach ASI without qualitative paradigm shifts.Compute-optimal training improves performance, while some apparent capability discontinuities may be metric artefacts rather than genuine step-changes.
- Scaling AGI: Scaling faces a data wall as high-quality text may be exhausted later this decade, making it likely that progress beyond AGI must transcend human-generated data limits.Recent corpora have reached three trillion tokens through filtering and deduplication, but larger models are growing faster than novel text.
- AI paradigm shifts: Paradigm-shift pathways include test-time scaling, adaptive computation, unbounded retrieval-based working memory, latent imagination, learned-model planning, and diffusion-based decision-making.These approaches can expand capabilities beyond static model scale and fixed context windows, but true paradigm shifts are intrinsically difficult to forecast.
- Recursive improvement: Recursive improvement could accelerate capability gains dramatically, but it may fizzle out or require rapidly exploding resources, while physical experiments, manufacturing, and larger trials impose speed limits.AI is nevertheless likely to accelerate AI R&D through additional or faster instances, even before AGI.
6. Remarks
The remarks conclude that compute scaling could theoretically produce ASI, but practical progress likely depends on qualitative innovations, better benchmarks, and multi-agent scaling. They also emphasize that many ASI capabilities remain difficult to predict and that agency, creativity, and progress require careful conceptual distinctions.
- Multi-agent scaling: AGI collectives could scale rapidly by adding instances, divide complex problems among agents, and organize work through forms such as corporations, markets, and other groups.Back-of-the-envelope estimates place AI population scaling at about 25× per year, depending on compute growth.
- Scaling and innovation: Compute scaling is theoretically sufficient for ASI, but naive algorithms would require prohibitive compute growth, making sophisticated inductive biases and priors practically necessary.These can be incorporated through model architectures, training processes, scaffolding, or general datasets.
- Capability prediction: Predicting specific ASI capabilities is currently unresolved because performance at a given computational cost is computationally irreducible for many problems and incomputable for universal hypothesis classes.The report therefore treats claims about curing diseases, achieving fusion, or unifying physics as unanswered today.
- Evaluation: A major evaluation challenge is designing ASI benchmarks that measure general capabilities beyond human-level saturation with little human input, alongside measuring multi-agent scaling laws.These benchmarks are intended to support accurate tracking of progress.
- Creativity: Transformative artistic creativity would require more than cognitive or optimization power because artistic value is shaped by subjective social systems involving artists, audiences, critics, and cultural institutions.This contrasts with scientific value, which is often grounded in predictive power and empirical truth.
- Agency: Superhuman cognitive capability need not require an agentic architecture, but economic and practical pressures to reduce oversight may favor integrating powerful oracle-like capabilities into autonomous agents.Oracles interacting with a persistent world can themselves function as agents through text output, with reduced controllability and action bandwidth.
7. Outlook: Plenty That Needs To Be Done
Post-AGI technological trajectories, pathways, and frictions remain highly uncertain and require sustained interdisciplinary research, quantitative forecasting, and broader societal-impact mapping. Progress may not stall at AGI: collective scaling or recursive self-improvement could enable a smooth or potentially rapid transition into ASI within the next decade or two.
- Research priorities: Post-AGI pathways and bottlenecks are incomplete, highly uncertain, and should be treated as open research programs requiring future updates.The report calls for further research and sharper, more formalized questions.
- Research priorities: Research should map advanced AI’s significant societal impacts alongside technological progress, with global and massively interdisciplinary participation.The report highlights the need to study impacts across many aspects of society, not only post-AGI technological development.
- Bottlenecks and Frictions for Scaling: Priority questions concern scaling frictions, including data availability, compute-intelligence relationships, paradigm shifts, economic viability, harder AI research, embodied experimentation, and abstraction barriers.These questions examine whether generated data is useful, how quantitative and qualitative scaling trade off, and how physical or conceptual limits constrain capability growth.
- Quantitative Forecasting: Quantitative forecasting should connect effective compute, AI capabilities, and macroeconomic effects using measurable macro-quantities, coupled models, ensembles, and scenario simulations.Candidate quantities include cost per FLOP, compute efficiency, and sector-specific AI productivity.
- Recursive Improvement Dynamics: Recursive improvement is a major source of forecasting uncertainty because AI could improve architectures, optimizers, training data, or test-time performance.The report characterizes recursive improvement as potentially among the largest accelerators of AI progress while remaining poorly understood.
- Outlook: Within the next decade or two, progress could plausibly continue beyond AGI toward ASI, with collective scaling enabling further capabilities and recursive self-improvement potentially making the transition rapid.This conclusion assumes human-level AGI is reached and does not rule out an intelligence explosion; alignment remains an important, difficult working assumption.
AI Use
The document was predominantly written by humans, with language models used for limited wording and drafting assistance as well as broader editorial and research-support tasks.
- AI Use: More than 90% of the document was written from scratch by humans, while language models assisted with wording, drafting, structure, reviews, literature searches, and bibliography cleanup.Language-model involvement covered less than 10% of the manuscript.
A. Summary
The report examines pathways from human-level AGI to ASI, defined as general superhuman intelligence exceeding large groups of human experts. It identifies four non-exclusive routes while emphasizing major uncertainties, bottlenecks, and the need for interdisciplinary research.
- Definitions: ASI is defined as general superhuman intelligence that outperforms large groups of thousands of human experts working over years.AGI denotes at least median human performance across a very broad set of cognitive tasks.
- Pathways: Four potential pathways are scaling compute, models and data; algorithmic paradigm shifts; recursive improvement; and ASI emerging through group agent formation.These pathways are not mutually exclusive, so progress could occur simultaneously and produce compounding increases in AI capabilities.
- Group agent formation: Large agent groups may generate ASI, but orchestration and bureaucracy increase with group size, while humans may struggle to steer or consume their output.The report highlights the need to study organization, problem structure, and meaningful human interaction with very large, very fast agent groups.
- Scaling and bottlenecks: Scaling currently appears the most promising pathway, but economic, hardware, resource, data, and paradigm limits may constrain continued progress.Internet-scale data sources are nearing exhaustion, while synthetic and interactive data generation may not scale sufficiently to meet demand.
- Algorithmic paradigm shifts: Algorithmic paradigm shifts could overcome scaling limits, but recognizing and reaching viable new paradigms may require substantial research, investment, and technological integration.Research should advance paradigm-agnostic understanding and clarify how practical limits relate to fundamental limits.
- Recursive improvement: Recursive AI improvement could create self-accelerating capability growth, although training, experimentation, hardware, compute, energy, investment, and diminishing returns may dampen it.The report recommends monitoring how much AI facilitates AI research and developing recursive-improvement scaling laws.
B. Glossary
The glossary defines the report’s central intelligence benchmarks—AGI, ASI, and Universal AI—and introduces mechanisms, bottlenecks, and methods used to analyze progress between them. It also specifies formal concepts underlying universal prediction and machine intelligence.
- Core intelligence concepts: AGI (Minimal) denotes roughly median human-level intelligence on cognitive tasks, or a “Competent AGI”.
- Core intelligence concepts: ASI denotes a system exceeding large, well-coordinated human-expert collectives across virtually all domains.
- Bottlenecks and methods: A data wall occurs when model-size growth outpaces global production of novel, high-quality training data.
- Progress mechanisms: Group agency describes a “super-agent” emerging from orchestrated or self-organized interaction among numerous AGI agents or sub-agents.
- Progress mechanisms: Recursive improvement covers AI systems improving next-generation AI through automated AI R&D or better training data, potentially producing an intelligence explosion.
- Core intelligence concepts: Universal AI is the endpoint of the machine-intelligence continuum and is formally described by AIXI, the theoretical upper bound for machine intelligence.AIXI is optimal on average over all computable environments and tasks.