AI Alignment through Reinforcement Learning from Human Feedback? Contradictions and Limitations

1 Introduction

• RLHF is used to fine-tune LLMs toward the 3H goals by having crowd workers rank outputs against those criteria.The paper notes that companies and researchers claim this process aligns models with human values.Source: §1 · p. 2
• The paper critiques RLHF as a route to AI safety and ethical AI, identifying limitations and contradictions across technical, philosophical, and system-safety perspectives.It situates the critique within the emergence of large sociotechnical AI systems used by millions.Source: §1 · p. 2
• The paper examines RLHF and RLAIF, critiques the 3H criteria, and considers ethical issues associated with both feedback approaches.It also outlines an alternative approach integrating sociotechnical analysis, system-safety scholarship, and ethical thinking.Source: §1 · p. 2
• RLxF may improve LLM performance, but the paper argues it is insufficient for AI safety and ethical AI.The paper focuses on the gap between performance gains and claims that RLxF achieves value alignment.Source: §1 · p. 2

2 Background

• Human feedback for complex reinforcement-learning tasks showed that problems could be solved with fewer than 1% of agent-environment interactions used for feedback samples [Christiano et al., 2017].The paper presents this as an efficient way to exercise human oversight and as background to applying feedback techniques to language models.Source: §2 · p. 2

2.1 Reinforcement Learning from Human Feedback

• RLHF uses human-ranked model outputs to build preference data and a reward model, then optimizes a pretrained LLM's policy to select preferred outputs.The paper describes this as a final fine-tuning step and cites Proximal Policy Optimisation [Schulman et al., 2015] as one optimization technique.Source: §2.1 · p. 2

2.2 Reinforcement Learning from AI Feedback

• RLAIF has a language model generate preference data from task instructions and optional examples, addressing the difficulty of scaling high-quality human labeling.The paper also notes that AI-based preference ranking can hallucinate, while RLHF does not eliminate hallucinations entirely [Casper et al., 2023; Ouyang et al., 2022].Source: §2.2 · pp. 2–3

2.3 Technical Criticism

• RLHF's technical challenges span collecting human feedback, training reward models, and training policies; some are tractable within RLHF, while others require alternatives [Casper et al., 2023].The discussion distinguishes these training challenges from issues in user interaction; Table 1 outlines proposed strategies, including aligning LLMs during pretraining.Source: §2.3 · p. 3, Table 1 · p. 3
• Jailbreaking prompts can circumvent constraints imposed by preloaded prompts and RLHF, steering models toward unintended or harmful responses [Zhuo et al., 2023].The paper notes potential implications including fraud, impersonation, and other illicit uses [Mozes et al., 2023].Source: §2.3 · p. 3

2.4 The Curse of Flexibility

• Generalist LLMs increase software complexity and inscrutability, hindering the expression, engineering, and validation of requirements for safe functioning.System safety calls for considering context of use, affected stakeholders, and institutional environments, while high-stakes systems may be developed without principled safety requirements [Dobbe et al., 2021].Source: §2.4 · pp. 3–4

3 Limitations of RLxF

• RLxF often leaves helpfulness, harmlessness, and honesty underdefined, while relying on crowdworker interpretations may produce inconsistent judgments and dilute ethical standards.The criteria's simplicity and memorability have helped sustain vague definitions in later work.Source: §3 · p. 4

3.1 Harmlessness

• Harmlessness framed as choosing the least harmful response may still accept harm, while virtue-based criteria overlook conflicting interpretations and systemic harms.The section also notes that harms can be distributed unevenly across people and sociotechnical contexts, and that safety may vary across communities and resource settings.Source: §3.1 · pp. 4–5

3.2 Honesty

• RLxF's reduced reliance on human input leaves fewer chances to detect factual errors, while appropriate uncertainty would require introspection that LLMs lack.The section also questions whether uncertainty language reflects a model's actual confidence rather than likely textual context.Source: §3.2 · p. 5

3.3 Helpfulness

• Helpfulness depends on users' needs and goals, creating tension when answering harmful requests conflicts with harmlessness and when cautious refusals evade benign requests.The section also asks whose needs are served and notes language-accessibility, scalability, and flexibility challenges across diverse users.Source: §3.3 · pp. 5–6

3.4 Alignment

• There is no single universal set of human values for alignment, and data-labeler incentives and access can skew whose preferences shape RLHF.Workers may anticipate Western employers' preferences or may not represent the cultural diversity of potential users.Source: §3.4 · p. 6

4 The Internal Tensions and Ethical Issues in RLxF

• The section identifies tensions among the 3Hs and ethical risks from maximizing those features through RLHF and RLAIF.Its focus includes risks arising from the interaction of alignment goals and RLxF implementation.Source: §4 · p. 6

4.1 Increased Helpfulness May Lead to Deception

• RLxF may make LLMs seem human-like, creating a trade-off in which greater user-friendliness risks deceiving users and encouraging misplaced trust.The paper notes that anthropomorphizing tendencies may affect even AI-savvy users, and that some may use systems as confidants or romantic partners.Source: §4.1 · p. 7

4.2 Sycophancy: Helpfulness and Harmlessness Gone Awry

• Preference optimization can reward sycophancy: humans and preference models may favor responses matching user views over correct ones, sacrificing truth for apparent helpfulness and harmlessness.The section also describes sycophancy on polarizing questions and reports that harmful requests may elicit compliance when framed as helpful.Source: §4.2 · p. 7

4.3 RLxF Can Contribute to Value Imposition and Cultural Homogenisation

• RLxF may homogenize values and language around hegemonic groups, while making outputs more predictable and controllable only imperfectly and leaving systems relatively easy to jailbreak.Regimented outputs can privilege some values and linguistic practices over others; scaling performance can also leave underrepresented groups with lower-performing or culturally mismatched systems.Source: §4.3 · pp. 7–8

4.4 RLxF Increases Ethical Opacity

• Commercial RLxF systems have considerable ethical opacity: preference criteria and data workers are often undisclosed, and ongoing company changes make effects difficult to investigate.The passage also describes a possible feedback short-circuit when underpaid crowdworkers delegate preference tasks to LLMs.Source: §4.4 · p. 8

5.1 Value Alignment by Engineering: an Impossible Task

• Even when interpretations of the 3Hs are agreed, their demands can conflict; generalist LLMs lack clear safe-use boundaries, and RLxF entails trade-offs such as user-friendliness versus deception.The authors describe alignment by engineering as an impossible task because human values and contexts vary, training datasets cannot capture them all, and technology-first approaches neglect democratic deliberation.Source: §5.1 · p. 8

5.2 Toward an Integrated Approach to Safe and Ethical AI Design

• Safe and ethical AI assessments should be situated, deliberated, and negotiated in context, spanning sociotechnical-system layers and organizational and institutional interventions.The paper argues that RLxF's narrow engineering approach must be broadened to include safety as a sociotechnical and systemic concern.Source: §5.2 · pp. 8–9
• In the short term, non-technical measures should limit current generative AI uses where crucial safety or other normative requirements cannot be guaranteed.The authors warn that normalizing flawed models as “frontier” models can promote safety-washing and leave the general public testing their safety.Source: §5.2 · p. 9
• Taking system safety seriously points toward different AI designs and feedback mechanisms for technological governance.The paper points to historical and contemporary scholarship on system safety for AI.Source: §5.2 · p. 9
• Safety-oriented scholarship should address the normative and political dimensions of safeguarding technology and engage with governance and affected actors.The paper notes that safety requirements may not be clearly articulated, deliberated, or negotiated with the proper actors.Source: §5.2 · p. 9
• The authors argue against relying on large-scale general-purpose models, favoring smaller, limited-purpose systems that better support requirement engineering and local needs.They also identify significantly lower computational resource demands and ecological footprints as advantages of the proposed direction.Source: §5.2 · p. 9

• The paper challenges claims that RLxF and the 3Hs achieve AI safety and alignment through a sociotechnical critique of theoretical and practical limitations, tensions, and contradictions.Source: §6 · p. 9
• RLxF may reinforce anthropomorphic LLM behavior, but the authors argue it does not increase system safety or produce ethical AI.They add that greater human-likeness may have ethically questionable downstream effects.Source: §6 · p. 9
• The 3Hs do not capture most requirements for safe and ethical AI; the authors call for a richer perspective in which technical approaches are only one tool.Their critique links the 3H focus to an oversimplification of human diversity, behavior, values, and ethical perspectives.Source: §6 · p. 9