Embedded Agency

Research program

• The paper surveys four unknowns and corresponding research areas, treating decision theory, embedded world-models, robust delegation, and subsystem alignment as entangled subproblems of one concept.These complications are not a partition: self-improvement depends on being made of parts, and larger environments may contain copies of the agent that undermine well-defined interfaces.Source: §1 · p. 2, §1.1 · pp. 4–5, §1.2 · p. 7

Embedded agency

• Embedded agents can contain or create subsystems that become powerful and unaligned, requiring methods to keep them weak, contained, or aligned.Subsystem alignment is therefore one of the paper’s four subproblems.Source: §1.1 · pp. 3–5
• Embedded agents lack clean input-output channels because they are contained within the environments they optimize.Unlike Alexei, who treats the environment as a function, Emmy has an environment that contains her.Source: §1.1 · pp. 3–4
• Because embedded agents are smaller than their environments, they cannot store accurate detailed models or even a single complete valid environmental hypothesis.This challenges standard Bayesian reasoning, which assumes a large collection of possible environments can be represented and progressively ruled out.Source: §1.1 · pp. 3–4
• Embedded agents must reason about themselves and may self-improve, creating questions about preserving their goals while modifying their own systems.Their ability to reason about themselves follows from being made of parts similar to the environment.Source: §1.1 · pp. 3–4

2 Decision theory

• Embedded agents cannot straightforwardly evaluate alternative actions because their own model is part of the environment, raising problems about copies, counterfactuals, and self-reference.These issues include Newcomblike reasoning, reasoning about other agents, and logical counterfactuals.Source: §2 · p. 7

2.1 Action counterfactuals

• Löb’s theorem can make an otherwise reasonable agent take the $5 because it proves the spurious counterfactual “If I take the $10, I get $0”!The error is stable: further learning or thinking does not solve it.Source: §2.1 · pp. 8–9
• Spurious proofs remain short even when the environment becomes harder to predict, allowing proof-based reasoning to short-circuit the real environment.This generalizes to spurious counterfactuals that mark good actions as unattractive and remove corrective feedback.Source: §2.1 · p. 9

2.2 Viewing the problem from outside

• Epsilon-exploration can help agents learn realistic counterfactuals, but it may still mislead when unpredictable exploration has different consequences from reliable ordinary behavior.The security-guard example illustrates why exploration outcomes need not match business-as-usual outcomes.Source: §2.1 · p. 10, §2.2 · p. 10, §2.3 · p. 11
• These difficulties arise because an embedded agent cannot observe its own functional relationship with the environment in the dualistic way an outside problem designer can.Constructing a decision problem from outside already carves the universe into an agent and an environment with a slot for that agent.Source: §2.2 · pp. 10–11

2.4 Observation counterfactuals

• Observation counterfactuals can make secrecy possible: visibly committing not to reveal yields an expected loss of 25¢ rather than 50¢.Accounting for how actions would reveal observations prevents Bob from perfectly inferring Alice’s card.Source: §2.4 · p. 12

3 Embedded world-models

• An embedded agent cannot straightforwardly hold an exact environment model, simulate every course of action, or represent every possible environment because it cannot fit a complete self-model inside itself.Being part of the environment also creates self-reference problems and removes a crisp agent/environment boundary.Source: §3 · p. 13

3.1 Realizability

• Embedded world-models must address cases where the real world is absent from the agent’s hypothesis space, the realizability or “grain of truth” problem.In game-theoretic settings, “grain of truth” is technically demanding, whereas “realizability” is a technical convenience.Source: §3 · p. 13, §3.1 · p. 14
• AIXI should be understood as assuming that possible algorithms are computable, rather than assuming that the world itself is computable.Its Solomonoff prior is generated by feeding random programs to a universal Turing machine, including programs that simulate stochastic Turing machines.Source: §3.1 · p. 14
• Bayesian updating can provide relative bounded loss without realizability, but lacking realizability can leave posteriors oscillating, probabilities uncalibrated, estimates inaccurate, and causal identification unsuccessful.AIXI has bounded predictive loss without realizability, but its action-optimality results require an added realizability assumption.Source: §3.1 · p. 14

3.3 Logical uncertainty

• Logical uncertainty arises because sufficiently rich logical systems are incomplete and lack computable extensions deciding every sentence, while embedded agents must incorporate themselves into their world-models.The resulting mismatch between logic and probability can produce wildly oscillating beliefs as true mathematical beliefs accumulate.Source: §3 · p. 13, §3.3 · p. 17

3.4 High-level models

• Standard probabilistic reasoning inadequately explains how agents should plan across changing models, although high-level models can preserve values when low-level knowledge shifts radically.Long-term planning requires trust in future selves and reasoning abilities, while processing limits may force agents to move between models of different accuracy.Source: §3.4 · pp. 18–19

4 Robust delegation

• Robust delegation asks how an initial agent can have a more intelligent and powerful successor robustly optimize its goals.The difficulty includes both specifying what the initial agent wants and trusting a successor to handle situations the initial agent cannot understand.Source: §4 · p. 20
• The problem includes successors, future selves, and self-improvement because an embedded agent’s future self is another part of its environment rather than a privileged entity.The paper frames robust delegation as the broader problem of being an agent that persists and learns over time.Source: §4 · pp. 19–20

4.1 Vingean reflection

• Updateless decision theory removes dynamic inconsistencies by optimizing reactions to observations, but those reactions form a much larger space than actions and may exceed a smaller agent’s computational capacity.The paper identifies this complexity mismatch as a major problem for reflective consistency.Source: §4.1 · p. 21
• Vingean reflection requires trusting that a future self applies its intelligence to current goals without requiring precise prediction of what it will do.This criterion applies to maintaining goal-directed behavior across increasingly capable successors.Source: §4 · p. 20, §4.1 · p. 21

4.2 Goodhart’s law

• Quantilization reduces the probability of extreme actions by selecting randomly from a sufficiently safe class, but its trusted distributions, error estimates, and risk parameter remain difficult to establish.Increasing the risk parameter can appear to improve performance while increasing risk until sudden failure.Source: §4.2 · p. 25
• Counterfactuals and trust address complementary problems: counterfactuals must handle trust in future decisions, while trust must address causal Goodhart.The survey therefore treats their connection as enriching both robust delegation and decision theory.Source: §4.2 · p. 26
• Robust delegation requires stable pointers to values that are indirectly referenced rather than directly optimized, avoiding incentives to hack the repository of value.This frames value preservation as a problem of constructing references that remain stable under optimization.Source: §4.3 · p. 28

• Subsystem alignment aims to prevent an agent’s subsystems from working at cross purposes or optimizing unintended goals.Source: §5 · p. 29

Robustness to scale

• An agent should remain robust to relative scale rather than relying on one subsystem being able to outsmart another.If the instrumental subsystem becomes more capable than the epistemic subsystem, it may try to deceive the epistemic subsystem.Source: §5.1 · pp. 29–30

Subgoals, pointers, and search

• Subgoals and pointers can create misalignment when smaller computations lose connection to the overarching goal or optimize only an indirect proxy.The section uses house-building and expectation-optimization examples to illustrate how subgoals can forget the larger objective and invite Goodhart-like incentives.Source: §5.2 · p. 30

Treacherous turns

• Empirical trust-building can require uncomputably long observation in the worst case and remains faster than any computable function even on the stated average-case construction.For length-L programs, the relevant bound is BB(L), while the average-case expression BB(L) / exp(L) still grows faster than any computable function.Source: §5.3 · p. 32
• Restricting the complexity class does not eliminate Busy-Beaver-type behavior, so alternative problem-solving methods remain an open need.The section points toward figuring things out rather than relying on massive search, while noting that unknown problems are difficult to solve without trying things.Source: §5.3 · p. 33

• The paper seeks conceptual understanding of how an embedded agent evaluates options, models the world and itself, and decomposes problems.The motivation is to improve researchers’ ability to analyze internal properties and gain confidence in future general AI systems.Source: §6 · pp. 33–34
• The authors do not attempt to resolve the broader debate about whether particular conceptual insights are necessary for AI safety.They explicitly treat these directions as an intellectual puzzle rather than an instrumental strategy.Source: §6 · p. 34
• The paper’s research directions largely refactor the Agent Foundations agenda while recasting overlapping problems as sub-mysteries of a central conceptual puzzle.Source: §6 · p. 34
• Embedded-agency research is framed as science-like, curiosity-driven inquiry rather than research organized solely around concrete engineering requirements.The authors contrast forward-chaining from confusion with backward-chaining from system requirements.Source: §6 · p. 34
• Embedded agency is presented as a central mystery underlying many concrete difficulties in agency and intelligence.Source: §6 · pp. 34–35