Source-linked AI summary
Future progress in artificial intelligence: A survey of expert opinion
Vincent C. Müller, Nick Bostrom
TL;DR
The paper asks how experts assess the timing, development, and risks of high-level machine intelligence, amid uncertainty and disagreement about these predictions. It uses a brief questionnaire distributed to four expert groups and finds substantial probability assigned to HLMI and subsequent superintelligence, with meaningful concern about harmful outcomes.
Problem
The study addresses limited and potentially biased evidence about expert views on when HLMI may arise, how quickly superintelligence could follow, and its likely impact.
Method
The authors designed a brief questionnaire and distributed it online to four expert groups, while testing research approaches, timelines, transition speed, and impact assessments.
Results
A 50% HLMI probability was placed at 2040 overall, experts saw significant probability of superintelligence within 30 years thereafter, and assigned 31% probability to a bad or extremely bad outcome.
Takeaways & Limitations
The authors conclude that expert views warrant investigating superintelligence’s potential impact and risks before its development.
Takeaways & Limitations
The questionnaire’s concepts are difficult to formulate because intelligence and progress are unclear, while human-level intelligence is elusive and can provoke resistance.
Abstract
from arXiv · showhide
There is, in some quarters, concern about high-level machine intelligence and superintelligent AI coming up in a few decades, bringing with it significant risks for humanity. In other quarters, these issues are ignored or considered science fiction. We wanted to clarify what the distribution of opinions actually is, what probability the best experts currently assign to high-level machine intelligence coming up within a particular time-frame, which risks they see with that development, and how fast they see these developing. We thus designed a brief questionnaire and distributed it to four groups of experts in 2012/2013. The median estimate of respondents was for a one in two chance that high-level machine intelligence will be developed around 2040-2050, rising to a nine in ten chance by 2075. Experts expect that systems will move on to superintelligence in less than 30 years thereafter. They estimate the chance is about one in three that this development turns out to be 'bad' or 'extremely bad' for humanity.
1. Introduction
The paper examines expert expectations about high-level machine intelligence and possible subsequent superintelligence, amid disagreement about whether these developments are imminent or science fiction.
- General AI began as a vision of machines simulating every aspect of learning and intelligence, but most current AI research focuses on specific problems.
- Superintelligence is defined as intellect greatly exceeding human cognitive performance across virtually all domains of interest.
- A proposed route to superintelligence is recursive improvement, in which human-level artificial general intelligence creates systems with still-higher intelligence.
- The study asks experts when high-level machine intelligence may arrive, whether it may lead to superintelligence, and what consequences could follow.
2. Questionnaire
The questionnaire surveyed invited experts across four groups, using concise, comparable questions to elicit views on HLMI and its possible development. The authors also addressed question-design challenges, prior surveys, and potential respondent-selection bias.
- 2.1. Respondents: Approximately 550 participants from four groups received individualized online invitations, reminders, and a questionnaire link.
- 2.1. Respondents: Questionnaire authorship, conference familiarity, and differing knowledge of the researchers contributed to varying response rates across groups.
- 2.1. Respondents: The groups differed in theoretical and ideological backgrounds, including theory-oriented critics of rapid AI progress and technically focused AGI researchers.
- 2.1. Respondents: Researchers handled overlapping group membership by sending one questionnaire per individual while counting responses for each applicable group.
- 2.3. Methodology: HLMI was defined behaviorally as a system able to perform most human professions at least as well as a typical human.
- 2.3. Methodology: The survey used four main questions and three respondent questions, with simple choices intended to improve response rates and enable comparisons with earlier questionnaires.
- 2.3. Methodology: The authors primed respondents with possible research approaches before asking about HLMI timelines, partly to examine whether preferred approaches correlated with predictions.
- 2.4. Prior work: The study adapted questions from earlier expert surveys because previous efforts used small or specific samples and sometimes had methodological problems.
3. Questions & Responses
Respondents evaluated approaches to HLMI, estimated when it might arise, assessed the speed of transition to superintelligence, and judged its potential impact on humanity. Results indicate substantial confidence in eventual HLMI and superintelligence, alongside uncertainty about timing and consequences.
- 3.1. Research Approaches: Whole brain emulation received 0% among TOP100 respondents but 46% among AGI respondents, with no other significant group differences reported.
- 3.1. Research Approaches: No relevant correlations were found between selected research approaches and subsequent HLMI timeline predictions.
- 3.2. When HLMI?: Respondents selected years for 10%, 50%, and 90% probabilities that HLMI would exist, assuming scientific activity continued without major negative disruption.
- 3.2. When HLMI?: 2040 was the overall median year for a 50% HLMI probability, compared with a mean of 2081 because later outliers and ‘never’ responses extended the distribution.
- 3.3. From HLMI to superintelligence: Experts assigned low probability to superintelligence within 2 years after HLMI but a significant probability within 30 years.
- 3.4. The impact of superintelligence: Impact probabilities were reported as means, with a notable difference between theoretical groups PT-AI and AGI and technical groups EETN and TOP100.
- Respondent characteristics: Respondents rated their expertise at a mean of 5.85 for the questionnaire topics and 6.26 for technical AI work.
4. Evaluation
The evaluation examines possible respondent-selection bias and summarizes experts’ estimates about HLMI, superintelligence, and its consequences. The authors caution that the results gauge perceptions rather than provide well-founded predictions.
- Selection-bias in the respondents?: The study tested whether non-respondents were biased toward later HLMI estimates by obtaining additional responses from PT-AI and TOP100.The additional sample comprised one PT-AI respondent and two TOP100 respondents.
- Selection-bias in the respondents?: The additional responses provided no support for later-arrival bias, although their very small sample prevented confident judgment.The PT-AI respondent was earlier than the mean but later than the median; both TOP100 respondents were earlier than the mean and median.
- Lessons and outlook: The questionnaire’s raw data, basic results, comments, and an online version were made available through a companion site.The authors also suggested repeating the questionnaire longitudinally.
- Results: A 50% probability of HLMI was associated with 2040–2050, and a 90% probability with 2075.The authors describe these as the experts’ overall view, while stressing that the results should be taken with caution.
- Results: Experts estimated that superintelligence would follow HLMI within 2 years at the 10% level and within 30 years at the 75% level.These estimates concern the time from reaching human-level ability to greatly surpassing human performance.
- Lessons and outlook: The authors state that the study aimed to gauge perception, not produce well-founded predictions.They recommend that readers treat the results with caution and investigate the raw data for detailed conclusions.
- Results: 31% of experts estimated that superintelligence would be bad or extremely bad for humanity.The questionnaire distinguished between extremely bad outcomes, defined as existential catastrophe, and other response categories.
Appendix 1: Online Questionnaire
The online questionnaire was an invitation-only survey intended to capture researchers’ views on AI progress and its impacts. Responses were anonymized and the results were to be publicly available.
- Purpose: The questionnaire aimed to gauge how AI researchers view progress toward intelligent machines and the impacts of reaching those goals.It was directed toward researchers in artificial intelligence or AI theory.
- Administration: Participation was by invitation only, and unsolicited submissions were to be disregarded.
- Administration: Answers were anonymized, and results were scheduled for public release through the Programme on the Impacts of Future Technology.
A. The Future of AI
The questionnaire defines HLMI behaviorally and asks about research approaches, arrival dates, the speed of superintelligence, and long-run impacts on humanity.
- Definitions: HLMI is defined as a system able to carry out most human professions at least as well as a typical human.
- Research approaches: Respondents were asked which research approaches might contribute most to developing HLMI.Options included algorithmic complexity theory, computational neuroscience, neural networks, evolutionary systems, faster hardware, cognitive architectures, and other methods.
- Research approaches: The research-approach options also included integrated cognitive architectures and other methods currently known to at least one investigator.
- Forecasts: Respondents estimated years corresponding to 10%, 50%, and 90% probabilities that HLMI would exist.The question assumed that human scientific activity continued without major negative disruption.
- Superintelligence: Respondents estimated the probability that, after HLMI, machine intelligence would greatly surpass every human in most professions within 2 years or 30 years.
- Impacts: Respondents distributed probabilities across five possible long-run impacts on humanity, from extremely good to extremely bad.The extremely bad category was defined as existential catastrophe.
B. About you
The questionnaire collected respondents’ expertise, discipline, comments, and identifying preferences alongside basic anti-spam and invitation information. The invitation emphasized the survey’s purpose, brevity, anonymity, and public dissemination.
- Respondent information: Respondents were asked to describe their expertise in relation to the questionnaire and technical AI work.
- Respondent information: Respondents were asked to identify their main academic discipline, including biology, psychology, cognitive science, or another field.
- Comments: Respondents could submit a brief comment and choose whether their name would accompany it.The questionnaire stated that answers remained anonymous even if a comment was named.
- Form administration: The form included a CAPTCHA asking respondents to enter characters shown in an image.
- Invitation: The invitation targeted prominent AI researchers to assess views on progress toward intelligent machines and associated impacts.
- Invitation: The questionnaire contained four multiple-choice questions, three statistical questions about respondents, and an optional comments field.The invitation said completion would take only a few minutes.
- Invitation: The invitation encouraged responses even from people who considered the exercise futile or misguided.
- Invitation: Results were intended for use in Bostrom’s forthcoming book and public release, while answers remained anonymous.