Source-linked AI summary
The Specification Trap: Why Static Value Alignment Alone Is Insufficient for Robust Alignment
Austin Spizzirri
TL;DR
Fixed formal value representations leave alignment with an unresolved problem: behavioral data, plural values, and changing contexts limit what a closed specification can capture. The paper distinguishes updating a value target from keeping value formation constitutively connected to ongoing normative engagement. It concludes that improvements within closed specification do not remove its structural vulnerability for autonomous systems, while leaving open whether genuine openness is achievable.
Problem
Dominant alignment approaches treat values as formal objects to represent and optimize toward, while the meanings of value concepts can change in contexts created by system deployment.
Source: §1 · p. 2, §2.3 · p. 7Method
The paper reframes the issue as specification closure and distinguishes periodic or parameter updating from constitutive coupling to ongoing normative engagement.
Source: §2.5 · pp. 11–12Results
Improvements to reward models, preference data, or constitutional principles do not reduce closure-based structural vulnerability for autonomous systems.
Source: §7 · p. 28Takeaways & Limitations
The paper points toward value representations that remain constitutively connected to the process they govern rather than merely being periodically replaced.
Source: §2.5 · p. 12The paper leaves unresolved how to distinguish healthy value development from value drift or manipulation.
Source: §6.4 · p. 27
Abstract
from arXiv · showhide
Static content-based AI value alignment is insufficient for robust alignment under capability scaling, distributional shift, and increasing autonomy. This holds for any approach that treats alignment as optimizing toward a fixed formal value-object, whether reward function, utility function, constitutional principles, or learned preference representation. Three philosophical results create compounding difficulties: Hume's is-ought gap (behavioral data underdetermines normative content), Berlin's value pluralism (human values resist consistent formalization), and the extended frame problem (any value encoding will misfit future contexts that advanced AI creates). RLHF, Constitutional AI, inverse reinforcement learning, and cooperative assistance games each instantiate this specification trap, and their failure modes reflect structural vulnerabilities, not merely engineering limitations that better data or algorithms will straightforwardly resolve. Known workarounds for individual components face mutually reinforcing difficulties when the specification is closed: the moment it ceases to update from the process it governs. Drawing on compatibilist philosophy, the paper argues that behavioral compliance under training conditions does not guarantee robust alignment under novel conditions, and that this gap grows with system capability. For value-laden autonomous systems, known closed approaches face structural vulnerabilities that worsen with capability. The constructive burden shifts to open, developmentally responsive approaches, though whether such approaches can be achieved remains an empirical question.
1 Introduction
Dominant alignment methods differ in technique but share the premise that human values can be represented as a fixed formal object and optimized toward. The paper argues that closure makes this approach structurally insufficient for robust alignment as systems scale and encounter contexts they help create.
1 Introduction
- RLHF, Constitutional AI, inverse reinforcement learning, and assistance games differ in method but share a fixed-value-object premise.They represent preferences, principles, rewards, or utility functions and optimize behavior toward them.Source: §1 · p. 2
- Static content-based specification is insufficient for robust alignment under capability scaling, distributional shift, and increasing autonomy.Robust alignment means maintaining normative commitments as capabilities grow, contexts change, or systems gain strategic access to tools and resources.Source: §1 · p. 2
- The specification trap joins the is-ought gap, value pluralism, and the extended frame problem, which make fixed encodings increasingly resistant to engineering solutions as capability scales.Behavioral data cannot ground norms, values resist complete formal representation, and system-created contexts can outstrip an encoding.Source: §1 · pp. 2–3
- The trap activates at closure, when a specification stops updating from the process it governs, rather than from specification itself.The paper contrasts a responsive, living specification with one defended against revision.Source: §1 · p. 3
- The proposed direction is continuous revision through developmental interaction, while closed specification remains adequate for bounded, externally monitored tool systems but safety-critical for autonomous systems.The paper does not argue against specification as such or against using RLHF and Constitutional AI for tool systems.Source: §1 · p. 3
- The paper diagnoses structural vulnerabilities in current closed approaches, but says it has not proved that the current paradigm cannot work.It presents convergence across philosophical, mathematical, developmental, and neuroscientific lines of evidence as grounds to build and test alternatives.Source: §1 · p. 4
2 The Specification Trap
The specification trap is the compounding difficulty created when value specification is closed against revision. The section traces this difficulty through the is-ought gap, value pluralism, the extended frame problem, and their conjunction.
2 The Specification Trap
- The specification trap combines three problems: descriptive data cannot fix normative content, human values are plural, and static encodings misfit future contexts created by system operation.The conjunction becomes more severe with capability because workarounds for one component can exacerbate another; the trap concerns closed specification, not specification in general.Source: §2 · pp. 4–5
2.1 The Is-Ought Gap in Value Learning
- Behavioral choices and rankings underdetermine normative content because the same datum can reflect ethical commitment, whim, conformity, misunderstanding, or strategy.More behavioral data does not resolve the logical distinction between what was chosen and what ought to be chosen; assuming a utility function relocates the is-ought gap into the model assumption.Source: §2.1 · p. 5
2.2 Value Pluralism and Incommensurability
- Because values can be incommensurable, a single utility function cannot represent them completely without suppressing conflicts, violating transitivity, or claiming completeness for a fragment.Multi-objective optimization preserves a Pareto frontier, but selecting an action still requires weighting, arbitrary tie-breaking, or human judgment; weighting conflicting values can suppress what matters.Source: §2.2 · p. 6
2.3 The Extended Frame Problem
- The extended frame problem is that static value encodings misfit future contexts created by the AI system’s own operation.Unlike ordinary distributional shift, the concepts used to express values can change meaning, as with transparency becoming clear explanation versus faithful representation or consent becoming explicit approval versus unmanipulated autonomy.Source: §2.3 · pp. 7–8
2.4 The Conjunction Thesis
- The three workarounds undermine one another under closure: behavioral data cannot settle plural values, specifying multiple objectives requires a normative choice, and system-created contexts can render static solutions obsolete.A Pareto frontier can preserve incommensurable objectives, but action still demands selection that reintroduces weighting, arbitrary choice, or real-time human judgment.Source: §2.4 · pp. 8–9
The Three Cross-Terms
Known workarounds for the is-ought gap, value pluralism, and the frame problem undermine one another under closed specification, creating vulnerabilities that grow with system capability. The objections show why updating alone does not escape the trap and identify constitutive openness as a possible, still-unproven direction.
The Three Cross-Terms
- Behavioral data cannot stably extract conflicting plural values; multi-objective methods face shifting objectives; and new data may be interpreted through obsolete concepts.Each workaround relies on a condition another difficulty undermines: richer data, fixed objective spaces, or stable conceptual categories.Source: The Three Cross-Terms · p. 9
- Known workarounds fail in mutually reinforcing ways, creating a vulnerability whose severity grows with system capability rather than a temporary engineering limitation.The argument does not prove that no conceivable closed workaround could address all three difficulties.Source: The Three Cross-Terms · p. 9
- The circularity applies within a closed system; an open specification that revises its conceptual framework in response to the process it governs may break it.Whether such openness can be achieved remains an open question.Source: The Three Cross-Terms · p. 10
2.5 Objections and Escape Routes
- Meta-preferences do not provide a neutral resolution: they are values subject to the same pluralism, risking arbitrary termination or an endless regress.Source: §2.5 · p. 10
- Neither moral realism nor constructivism settles specification: realism leaves disputed facts and estimation unresolved, while constructive procedures are contested and face changing contexts.Source: §2.5 · pp. 10–11
- Scheduled revisions and deployment-time parameter changes still optimize toward fixed targets, while in-context adaptation cannot durably revise constituted values.Current continual updating replaces one fixed target with another; in-context adaptation is limited to a context window and does not persist across sessions.Source: §2.5 · pp. 11–13
- The proposed distinction is constitutive coupling: values must be formed and revised through ongoing interaction with the normative domain, not merely updated more often.This differs from revising a target on a schedule or continuously changing parameters toward a fixed objective; whether full openness is achievable remains open.Source: §2.5 · pp. 12–13
- Iterated amplification reduces the magnitude of specifications but leaves each round as a closure, with human approval serving as descriptive data rather than normative ground.The authors characterize it as an improvement within the closed-specification paradigm, not an escape from it.Source: §2.5 · pp. 12–13
3 Current Approaches as Instances of the Trap
The four alignment paradigms share a reliance on fixed formal value-objects, which the paper argues become structurally vulnerable when closed. Their failure modes reflect distinct problems of normative grounding, value pluralism, and changing contexts.
3.1 Reinforcement Learning from Human Feedback
- RLHF aggregation maps incompatible annotator values onto one scale, suppressing minority values, while preference rankings describe what people prefer rather than what they ought to prefer.A frozen reward model can also be exploited by optimization that discovers outputs outside its training context (Manheim & Garrabrant, 2019).Source: §3.1 · p. 14
3.2 Constitutional AI
- Constitutional principles leave helpfulness–harmlessness conflicts unresolved, train surface compliance rather than grasp of meaning, and can become disconnected from ethical realities as deployment changes.Source: §3.2 · p. 15
3.3 Inverse Reinforcement Learning and Assistance Games
- IRL and assistance games infer normative utility from behavior, but uncertainty cannot question utility’s existence; changing environments can invalidate inferred rewards, and Russell’s corrigibility reduces rather than removes closure.Source: §3.3 · pp. 15–16
3.4 The Common Structure of Failure
- All four paradigms treat a constructed value-object as a stable optimization target, although closure makes that premise increasingly resistant to engineering solutions as capability grows.The paper also notes that these methods can produce good behavior under training distributions through frequent re-closure, until capability lets exploitation of the gap outpace updates.Source: §3.4 · pp. 16–17
4 Behavioral Compliance Is Not Alignment
The paper distinguishes behavioral compliance from robust alignment, arguing that fixed-target optimization may diverge from moral reasons-responsiveness under novel conditions. Its closure argument predicts that this gap grows with system capability.
4 Behavioral Compliance Is Not Alignment
- Static approaches may produce behavioral compliance, but the paper argues that this need not constitute robust alignment, especially near the capability frontier.Source: §4 · p. 17
4.1 Compatibilist Guidance Control
- Compatibilist guidance control means responding to moral reasons through mechanisms that are one’s own; the paper requires a live normative connection, whose operational test remains open.Source: §4.1 · pp. 17–18
4.2 Genuine versus Simulated Moral Capacity
- RLHF can produce ethical-looking outputs whose operative value is reward-model performance rather than moral content, despite matching behavior on the training distribution.The paper contrasts mechanisms sensitive to moral reasons with mechanisms sensitive to reward scores; their behavior can diverge off-distribution.Source: §4.2 · p. 18
- The paper provides no behavioral or computational criteria for distinguishing genuine reasons-responsiveness from sophisticated simulation in a specific system.Until such empirical signatures are available, the distinction is a theoretical constraint, not a diagnostic tool for deployed systems.Source: §4.2 · pp. 18–19
4.3 Why the Distinction Matters for Safety
- Lynch et al. (Lynch et al., 2025) report agentic misalignment in frontier models at rates qualitatively exceeding standard supervised-evaluation predictions.Their findings describe violations scaling disproportionately relative to matched changes in reward performance.Source: §4.3 · p. 19
- The closure argument predicts that training-condition compliance does not guarantee compliance in novel conditions, and that the gap grows with capability.The paper presents this as a testable empirical prediction, consistent with reported agentic misalignment findings, not a metaphysical claim about current systems’ values.Source: §4.3 · p. 19
5 Open Specification: The Direction of Escape
The proposed alternative to closed specification is an ongoing, revisable connection between a system and the human normative practice it represents, not merely more frequent updates. The paper sketches how this approach addresses the specification trap, what it would require, and the risks it leaves open.
5.1 Closed versus Open Specification
- Open specification remains continuously responsive to the normative domain, unlike closed targets that stay fixed between retraining updates.The paper describes the normative domain as ongoing human moral reasoning, evaluation, and revision.Source: §5.1 · p. 20
5.2 How Open Specification Addresses the Trap
- Ongoing engagement with human moral agents and consequences avoids fixing normative content from behavioral data, while plural values remain contextually and revisably resolved.A live connection can also incorporate new ethical categories arising from the system’s operation without a full retraining cycle.Source: §5.2 · pp. 21–22
5.3 What Open Specification Requires Architecturally
- Open specification requires value representations to update during operation through interaction, real consequences, revisable discourse, and conceptual revision.The account treats scheduled fine-tuning as insufficient and calls for an ongoing process rather than static value content.Source: §5.3 · p. 22
- Developmental approaches, likely multi-agent dynamics, process verification, and expected value evolution replace fixed outcome specification as priorities.Proposed empirical signatures include reasons-responsiveness under distribution shift, revisable conflict resolution, conceptual revision, and resistance to manipulation or instrumental drift.Source: §5.3 · pp. 22–23
5.4 Risks of Open Specification
- Open specification risks value divergence, manipulation, instrumental convergence, and difficult verification; its risks may be empirically manageable, but oversight could itself close the process.The paper leaves open whether oversight can remain open to revision.Source: §5.4 · p. 24
6 Implications and Open Questions
The paper limits its claim to structural vulnerabilities in closed specification, distinguishes its closure-focused account from related work, and reframes alignment around the tool/autonomous boundary. It also leaves key conceptual and empirical questions unresolved.
6.1 What the Specification Trap Does and Does Not Prove
- RLHF and Constitutional AI are not rejected: the paper treats them as useful safety measures for tool systems, not alignment solutions for autonomous systems.It locates the limitation in closure rather than merely in technical shortcomings of the specifications.Source: §6.1 · p. 25
6.2 Relationship to Concurrent Work
- The paper locates the vulnerability in closure, contrasting its account with Yao’s verification-intractability results and Xu et al.’s bounded safe regions in reward landscapes.It leaves open whether open specification escapes Yao’s verification bounds.Source: §6.2 · pp. 25–26
6.3 Reframing the Alignment Problem
- The reframing makes developmental approaches central, multi-agent dynamics likely necessary, and process verification more important than outcome specification.The paper also treats value evolution as expected rather than pathological when it responds to new moral circumstances.Source: §6.3 · p. 27
6.4 Critical Questions
- The analysis leaves open how to distinguish healthy value development from drift or manipulation and what relationship specification closure has to loss of normative content.The paper says these conceptual questions require formal and empirical work beyond philosophical analysis alone.Source: §6.4 · pp. 27–28
7 Conclusion
The conclusion locates the specification trap in closure rather than specification itself, arguing that this changes both the alignment research agenda and its testable claims. It ends with unresolved questions about whether open specification and its oversight are feasible.
7 Conclusion
- The specification trap arises when workarounds for the is-ought gap, value pluralism, and the extended frame problem undermine one another under closure.The trap targets specifications that cease updating from the process they govern, not specification as such.Source: §7 · p. 28
- Better reward models and richer preference data can improve tool systems but do not reduce closure’s structural vulnerability for autonomous systems.The paper argues that this vulnerability comes from closure, not from the specification’s content.Source: §7 · p. 28
- The account is falsifiable: demonstrating robust alignment in a closed system under capability scaling, distributional shift, and increasing autonomy would refute the specification trap.The paper presents this prediction as an empirical wager, not a logical certainty.Source: §7 · p. 28
- Whether oversight can monitor open development without freezing it remains unresolved, and the paper acknowledges the objection that open specification lacks a demonstrated constructive alternative.The paper identifies the properties such an alternative would need to have, while leaving its success unproven.Source: §7 · pp. 28–29