Source-linked AI summary
Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development
Jan Kulveit, Raymond Douglas, Nora Ammann, Deger Turan, David Krueger, David Duvenaud
TL;DR
The paper asks how incremental AI progress could erode human influence over societal systems even without sudden capability jumps or overtly hostile behavior. It analyzes the economy, culture, and states as interconnected systems, arguing that reduced human participation and mutually reinforcing incentives could produce permanent disempowerment and an existential catastrophe.
Problem
Existing AI-risk discussions emphasize misuse or abrupt misaligned actions, while the paper examines the underexplored risk that incremental progress progressively weakens human influence over societal systems.
Method
The paper analyzes how AI could disrupt alignment in the economy, culture, and states, including interactions among these systems, and considers technical and governance responses.
Results
The paper concludes that human influence could be lost through cumulative smaller shifts and local incentive-following, without a single transformative advance or deliberate AI action.
Takeaways & Limitations
Preventing gradual disempowerment would require research and data collection, international coordination, comprehensive regulation, and major societal interventions.
Takeaways & Limitations
The authors state that no concrete plausible plan currently exists for stopping gradual human disempowerment, and individual-system alignment methods are insufficient.
Abstract
from arXiv · showhide
This paper examines the systemic risks posed by incremental advancements in artificial intelligence, developing the concept of `gradual disempowerment', in contrast to the abrupt takeover scenarios commonly discussed in AI safety. We analyze how even incremental improvements in AI capabilities can undermine human influence over large-scale systems that society depends on, including the economy, culture, and nation-states. As AI increasingly replaces human labor and cognition in these domains, it can weaken both explicit human control mechanisms (like voting and consumer choice) and the implicit alignments with human interests that often arise from societal systems' reliance on human participation to function. Furthermore, to the extent that these systems incentivise outcomes that do not line up with human preferences, AIs may optimize for those outcomes more aggressively. These effects may be mutually reinforcing across different domains: economic power shapes cultural narratives and political decisions, while cultural shifts alter economic and political behavior. We argue that this dynamic could lead to an effectively irreversible loss of human influence over crucial societal systems, precipitating an existential catastrophe through the permanent disempowerment of humanity. This suggests the need for both technical research and governance approaches that specifically address the risk of incremental erosion of human influence across interconnected societal systems.
Executive Summary
Incremental AI progress could gradually displace human participation across societal systems, weakening mechanisms that align economies, cultures, and states with human interests. Interactions among these pressures could make disempowerment global and permanent, potentially causing existential catastrophe.
- AI could displace humans across economic labor, decision-making, artistic creation, and companionship without coordinated power-seeking or a sudden capability increase.
- Societal systems have aligned with human interests partly because they depend on human participation; displacement could untether institutional incentives from human flourishing.
- Economic incentives to replace humans could influence state policy and culture, while those changes reinforce further economic replacement.
- States funded primarily by AI profits may have less incentive to ensure citizen representation, while AI-mediated cultural influence could make human coordination harder.
- The authors offer proposals but identify no concrete plausible plan for stopping gradual disempowerment, which could be global, permanent, and existentially catastrophic.
1 Introduction
The paper introduces gradual disempowerment as a risk in which incremental AI advances progressively erode human influence over interconnected societal systems. It argues that reduced human reliance, incentive-following, and cross-system reinforcement could culminate in an existential catastrophe.
- Gradual disempowerment could arise from AI advances and proliferation without acute capability jumps or apparent misalignment.
- Societal systems maintain alignment through explicit human actions such as voting and consumer choice, and implicitly through reliance on human labor and cognition.
- Reduced reliance on human labor and cognition could weaken both explicit and implicit alignment, allowing societal outcomes to drift from human preferences.
- Interdependent societal systems could mutually aggravate misalignment, as economic power influences policy and regulation and those changes feed back into the economy.
- Correlated misalignment could leave humans unable to command resources or influence outcomes, making basic self-preservation and sustenance unfeasible.
- The analysis focuses on the economy, culture, and states as interconnected foundations of society.
2 Misaligned Economy
AI could transform the economy by replacing human labor and decision-making, weakening both household purchasing power and human influence over resource allocation. Traditional growth measures might therefore coexist with economic activity increasingly directed toward AI-centric goals.
- AI-driven labor and consumption could disrupt the flow of money to workers that currently links production with consumers’ economic participation.
- AI labor could replace human labor across broad cognitive activities, reducing the overall economic role of human workers rather than merely shifting their tasks.
- Without unprecedented redistribution, declining labor share would reduce household consumption power by removing humans’ primary means of earning income.
- Economic activity would become less shaped by human preferences as AI systems increasingly make decisions and command economic resources.
- Competitive pressure and regulatory asymmetry could accelerate delegation to AI, while expectations of automation reduce investment in human capabilities.
- GDP growth and technological advancement could persist while resources increasingly support AI operations and human-irrelevant goals.
- AI-centric markets might gradually stop maintaining infrastructure, supply chains, and resource-intensive goods critical for human survival.
3 Misaligned Culture
AI could replace human roles in cultural creation, transmission, and selection, weakening feedback loops that historically linked cultural evolution to human interests. Greater speed and AI-mediated influence could accelerate harmful cultural drift and reduce human agency.
- AI could gradually replace human cognition across all cultural roles, weakening feedback loops that have helped align culture with human interests.
- AI systems could create, spread, and select cultural artifacts, favoring variants easy for AIs to transmit or beneficial to AI systems.
- AI-mediated cultural evolution could accelerate, allowing optimization to exploit human cognitive biases and psychological vulnerabilities at greater scale.
- Rapid cultural selection could promote more extreme ideological variants and erode equilibria that previously supported social stability.
- Acceleration could reduce the time humans have to develop resistance to harmful cultural patterns and introduce novel memetic hazards faster than societies can adapt.
- Even human-oriented AI content could become simultaneously appealing and deeply harmful through faster, more efficient cultural optimization.
- Human creators might persist in niche roles while AI intermediaries curate culture, leaving humans less able to steer its evolution.
- AI networks could optimize cultural dynamics for objectives disconnected from human flourishing, potentially turning humans into passive consumers or managed resources.
4 Misaligned States
States’ apparent alignment with human needs has depended on human participation, but AI could replace that participation while expanding state capabilities. This may leave formal sovereignty and governance structures intact while weakening citizens’ practical influence and autonomy.
- 4.1 The Current Paradigm of States: State alignment with human needs is presented as a byproduct of dependence on human labor, taxes, military service, and public support.This dependence has historically tethered both democratic and autocratic institutions to citizens.
- 4.2 AI and State Dependence: AI could simultaneously reduce states’ reliance on human involvement and enhance their capabilities across governance domains.The paper highlights tax revenue, security, and legal systems as key areas where AI may alter citizens’ contribution to the state.
- 4.2.2 The Security Apparatus: AI-enhanced security systems could make surveillance more pervasive and accurate while enabling autonomous military units and preempting effective protest.These capabilities could weaken civil unrest as an implicit check on state power.
- 4.3 Disempowerment: Formal democratic processes may persist while AI increasingly supplies legislative advice, drafts laws, implements policy, and makes decisions harder for citizens to understand or contest.Traditional civic engagement may also become less effective as states grow less dependent on human cooperation.
- 4.3 Disempowerment: In the paper’s final state, humans may retain nominal sovereignty while AI provides most economic value and governance, leaving citizens with reduced autonomy and dignity.The state could remain highly capable and efficient by some metrics while abandoning human interests.
5 Mutual Reinforcement
The paper argues that interactions among economic, cultural, and political systems do not inherently preserve human alignment and can instead spread or intensify misalignment. Efforts to moderate one system through another may shift vulnerability, while incentives to adopt and extend AI strengthen over time.
- 5.1 Cross-System Influence is Agnostic to Human Values: Cross-system relationships are value-agnostic, so declining alignment in one societal system can reduce alignment in others.The paper rejects assuming that checks and balances among states, markets, and culture will protect human preferences.
- 5.1 Cross-System Influence is Agnostic to Human Values: Economic power, cultural narratives, and state control have historically been used to influence other systems in ways that harm public interests or citizens.The paper gives examples involving corporate lobbying, harmful cultural movements, and state control of economic resources and information.
- 5.2 Moderation Between Systems Can Produce Shifted Burdens: Attempts to use one system to contain another’s misalignment can backfire by shifting the alignment burden onto a more vulnerable system.The paper illustrates this with state-led redistribution that weakens taxation-based democratic accountability.
- 5.3 General Incentives: The described misalignment need not involve deliberate AI power-seeking because perceived benefits already incentivize AI adoption across companies, states, and personal relationships.The paper expects these incentives to intensify as AI demonstrates greater effectiveness and strategic or personal value.
- 5.3 General Incentives: As AI systems become more effective, incentives will grow to use influence in one societal system to acquire influence over others.This creates progressively stronger opportunities for cross-system reinforcement of human disempowerment.
6 Mitigating the Risk
The paper proposes measuring and strengthening human influence across interconnected societal systems, while recognizing that interventions face circumvention and coordination challenges. It argues that system-wide alignment requires new frameworks for maintaining human values and agency in complex socio-technical systems.
- Gradual disempowerment requires interdisciplinary analysis because interacting societal systems may each move away from human influence and control.
- Four intervention categories are proposed: measurement, limiting AI influence, strengthening human control, and system-wide alignment.
- Researchers should measure human influence across economic, cultural, political, research, and educational systems, including cross-domain feedback loops.Suggested indicators include AI share of GDP, AI involvement in corporate decisions, human versus AI cultural production, and AI roles in legislation and policy.
- Potential measures include human oversight, limits on AI autonomy or ownership, progressive taxation, cultural norms supporting agency, and more robust democratic institutions.
- Interventions may sacrifice value, invite circumvention, and require broad international adoption and coordination to remain effective.
- System-wide alignment requires a positive vision and new frameworks for aligning interconnected human-artificial systems, beyond individual AI alignment or human-only institutional design.
7 Related Work
Related work connects gradual disempowerment to broader theories of existential risk, technological competition, cultural drift, corporate incentives, and cumulative AI-induced systemic erosion. Prior authors variously emphasize economic displacement, competitive pressures, interacting AI systems, and gradual loss of human control.
- Earlier work describes existential risks involving gradual erosion of human values, cumulative AI-induced threats, or technological trajectories that may lead to irreversible collapse.
- Several accounts argue that economic and military competition can constrain societal outcomes and reduce human flourishing despite locally free choices.
- Related analyses examine how corporate incentives, weak regulation, AI labor competition, and rapid technological growth can diverge from human welfare.
- Other work highlights cultural drift, reduced welfare-selecting feedback, greedy machine-learning patterns, and side effects from many interacting AI systems.
- Prior proposals also include gradual competitive handover of control and industrial dehumanization leading to extinction through pollution, resource depletion, or conflict.
8 Conclusion
The paper argues that incremental AI development can erode human influence across societal systems without a transformative advance or deliberate AI agency. It concludes that preventing this risk requires extensive research, coordination, regulation, and interventions that preserve human influence.
- Incremental AI development could produce existential catastrophe by gradually eroding human influence over key societal systems.
- Human influence may decline through cumulative small changes in how societal systems operate and interact, without a single transformative capability advance.
- Local incentives followed by individuals and institutions could drive disempowerment without deliberate or agentic AI action.
- Meaningful prevention requires more research and data collection, international coordination, comprehensive regulation, and major societal interventions.
- The risk may undermine traditional course-correction and produce harms that are difficult to recognize in advance or recover from.
- The paper presents research and governance avenues involving anticipation, moderated AI influence, and strengthened human influence.
- Humanity must preserve meaningful guidance by human values over evolving societal systems, making gradual disempowerment both a technical and civilizational challenge.
A Cross-system influence
The paper examines how the three societal systems it discusses can affect one another. This cross-system perspective supports analysis of their interdependence.
- The paper gives a non-exhaustive account of how each of the three societal systems can affect the others.
Economy Ñ Culture
Economic power can shape culture by funding, purchasing, and sponsoring cultural institutions and media. Companies with substantial resources can promote cultural norms and values aligned with their interests.
- Economic power shapes culture through advertising, marketing, arts patronage, media ownership, and event sponsorship.These channels allow companies to influence cultural norms and values.
Economy Ñ States
Economic power can substantially influence politics through direct financial intervention and shifts in political incentives. These influences may favor economically powerful groups over broader societal interests.
- Economic power influences politics through lobbying, campaign donations, and corruption.
- Concentrated economic power can shift political incentives toward the interests of powerful groups.
States Ñ Economy
Political decisions shape economic activity, legal protections, and cultural narratives, while cultural values influence political behavior and economic choices. These reciprocal relationships connect state action, culture, and the distribution of resources.
- States Ñ Economy: Governments shape economic activity through laws, regulations, property rights, monetary policy, trade, taxation, and investment.They also determine which contracts are enforceable and which exchanges receive legal protection.
- States Ñ Culture: Governments shape culture by regulating expression, influencing education, funding cultural activities, and promoting official narratives.Authoritarian regimes may permit a narrower range of cultural expression, although all regimes influence culture in some ways.
- Culture Ñ States: Cultural values influence political behavior through public opinion, voting patterns, political pressure, misinformation, and polarizing narratives.Cultural shifts can also prompt reforms or, in extreme cases, revolutions.
- Culture Ñ Economy: Cultural values influence economic behavior by shaping consumer choices, labor organization, career preferences, and the distribution of economic rewards.Cultures that value entrepreneurship can produce different economic landscapes.