Source-linked AI summary
Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback
Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, Scott A. Hale
TL;DR
As LLMs reach diverse users, aggregate alignment may not represent their differing preferences and values, while current approaches can reflect narrow provider and crowdworker judgments. The paper reviews these alignment concerns, develops a taxonomy of personalisation’s benefits and risks, and proposes three-tiered governance; it concludes that the taxonomy is provisional because effects depend on technology performance and user interaction.
Problem
Aggregate alignment may not represent the diversity of users’ preferences and values, while current alignment leaves unresolved whose preferences and values should guide models.
Source: §1 · pp. 1–2Method
Results
Takeaways & Limitations
The proposed framework aims to allow personalised alignment while keeping it within societal and organisational bounds.
Source: §1 · pp. 2–3The literature-based taxonomy may leave gaps, and its effects depend on system performance and user interaction, so it will require revision as technology changes.
Source: BENEFITS · p. 7
Abstract
from arXiv · showhide
Large language models (LLMs) are used to generate content for a wide range of tasks, and are set to reach a growing audience in coming years due to integration in product interfaces like ChatGPT or search engines like Bing. This intensifies the need to ensure that models are aligned with human preferences and do not produce unsafe, inaccurate or toxic outputs. While alignment techniques like reinforcement learning with human feedback (RLHF) and red-teaming can mitigate some safety concerns and improve model capabilities, it is unlikely that an aggregate fine-tuning process can adequately represent the full range of users' preferences and values. Different people may legitimately disagree on their preferences for language and conversational norms, as well as on values or ideologies which guide their communication. Personalising LLMs through micro-level preference learning processes may result in models that are better aligned with each user. However, there are several normative challenges in defining the bounds of a societally-acceptable and safe degree of personalisation. In this paper, we ask how, and in what ways, LLMs should be personalised. First, we review literature on current paradigms for aligning LLMs with human feedback, and identify issues including (i) a lack of clarity regarding what alignment means; (ii) a tendency of technology providers to prescribe definitions of inherently subjective preferences and values; and (iii) a 'tyranny of the crowdworker', exacerbated by a lack of documentation in who we are really aligning to. Second, we present a taxonomy of benefits and risks associated with personalised LLMs, for individuals and society at large. Finally, we propose a three-tiered policy framework that allows users to experience the benefits of personalised alignment, while restraining unsafe and undesirable LLM-behaviours within (supra-)national and organisational bounds.
1 Introduction
The paper argues that current LLM alignment can encode narrow preferences set by providers and small groups of crowdworkers, while explicit personalisation could better reflect individual users. It proposes mapping the benefits and risks of personalisation and governing it within national, supra-national, and provider-defined bounds.
1 Introduction
- Current alignment can reflect technology designers’ specifications and the preferences of small, often non-representative crowdworker groups.The paper describes this as implicit personalisation rather than a neutral representation of human preferences.Source: §1 · p. 2
- The paper defines personalised LLMs as learning from an individual end-user’s feedback to align with their preferences, values, or contextual knowledge.The scope includes adapting values, communication style, and retained user-specific information.Source: §1 · p. 2
- Personalisation may tailor assistance and include under-represented groups, but may also reinforce bias, essentialise identities, and narrow information diets.The paper raises the possibility that individual risks could accumulate into societal polarisation and weakened social cohesion.Source: §1 · p. 2
- The paper contributes a literature-grounded taxonomy of individual and societal benefits and risks, alongside a three-tiered governance framework.The framework assigns immutable restrictions to national or supra-national authorities, optional requirements to providers, and tailored requirements to end-users.Source: §1 · p. 2
2 Background
The background questions who defines alignment, what is being aligned, and whose judgments represent users. It presents explicit personalisation as a likely next step, supported by existing NLP work, feedback-learning techniques, and model customisation, while noting that a full review of personalisation literature is out of scope.
2.1 The Question of “Alignment”
- Alignment remains normatively unsettled: its meaning, target, and representative human reference group are all disputed.The paper distinguishes functional alignment from alignment to shared social values and morals, and notes distinctions among instructions, intentions, preferences, and values.Source: §2.1 · p. 3
- RLHF and related alignment practices may encode a prescriptive, narrow set of preferences from small, predominantly US-based crowdworker samples.The paper calls this implicit personalisation and notes that limited documentation can obscure who supplied the judgments.Source: §2.1 · p. 3
2.2 From Implicit to Explicit Personalisation
- Because human preferences and values vary, the paper argues that fully aligning models across populations may be futile and proposes explicit user-level personalisation instead.It describes this as micro-alignment to specific end-users’ preferences and values.Source: §2.2 · pp. 3–4
- Personalisation has precedents in internet technologies and NLP, but explicit user personalisation has had little integration with recent instruction-following and feedback learning in widely used public-facing models.Existing NLP work spans tasks including dialogue, summarisation, translation, and question answering.Source: §2.2 · p. 4
- The paper points to feedback-learning techniques and individual reward modelling as evidence that learning user-specific preferences is technically feasible.It also cites model customisation and branching as signs that adaptation is already occurring, while describing further personalisation as an expected development.Source: §2.2 · pp. 4–5
- Extensively summarising personalisation research is beyond the scope of this article.The authors say they are working on a separate review of implicit and explicit personalisation in NLP.Source: §2.2 · p. 4
3 A Taxonomy of the Benefits and Risks from Personalised LLMs
The paper builds a taxonomy of personalised LLMs’ benefits and risks at individual and societal levels, drawing on prior work across AI, LLMs, feedback learning, personalised NLP, and related internet technologies. It frames effects as paired trade-offs and describes the taxonomy as provisional because technology performance and user interaction remain uncertain.
3 A Taxonomy of the Benefits and Risks from Personalised LLMs
- The taxonomy distinguishes individual and societal effects and pairs potential benefits with corresponding risks where relevant.Examples include personalised utility paired with addiction or over-reliance, and individual bias reinforcement paired with societal polarisation.Source: §3 · p. 5, Table 1
- Its construction draws on existing risk taxonomies, RLHF and feedback-learning research, personalised NLP literature, and analogous internet-technology research.The analogies include recommender systems, automated influence, social media, content moderation, and the internet of things.Source: §3 · p. 5, BENEFITS · p. 7
- The societal categories include potential labour displacement and environmental costs from disaggregated training, storage, and inference.The taxonomy also lists workforce productivity as a potential benefit.Source: BENEFITS · p. 6
- The taxonomy is provisional: literature-based coverage may leave gaps, and effects depend on system performance and how users interact with the technology.The authors say future work will interview end-users, providers, and policy makers and revise the taxonomy as the technical landscape shifts.Source: BENEFITS · p. 7
I.B.1. Efficiency
Personalised LLMs may make task completion more efficient by predicting users’ intent or needs, reducing the prompts or inputs required. The section connects this proposed prompt efficiency to personalised search and feedback-based adaptation.
I.B.1. Efficiency
- Fewer prompts or inputs may be needed when a personalised model predicts users’ intent or aligns with their needs.The paper calls this potential gain “prompt efficiency.”Source: I.B.1. Efficiency · p. 7
- More accurate personalised search rankings are associated with greater search efficiency and less cognitive burden from irrelevant information.The paper uses this as an analogy for how personalised LLMs might help users reach desired outputs or complete tasks.Source: I.B.1. Efficiency · p. 7
- Prior work has used user feedback to adapt semantic relatedness and query intent, while the paper treats efficiency as inverse to decreased-labour quality-of-service harms.The cited framing links efficiency to a system operating more closely as intended.Source: I.B.1. Efficiency · p. 7
I.B.2. Utility
Personalised LLMs may be more useful by adapting to users’ intent, communication preferences and knowledge, and values. The proposed benefits include better support for diverse users and less reliance on values prioritised by majorities or technology designers.
I.B.2. Utility
- Personalised feedback may improve intent prediction because users both define tasks and rate outputs, and may better accommodate diverse linguistic styles and dialects.The authors also connect increased utility with reducing exclusion and narrowing performance differentials across a wider userbase.Source: I.B.2. Utility · pp. 7–8
- Adapting communication preferences and user knowledge may improve usefulness, with potential applications in education, healthcare, finance and legal contexts.Examples include tailoring responses to a learner’s knowledge, medical history, financial constraints or jurisdiction.Source: I.B.2. Utility · pp. 7–8
- Value personalisation may represent more diverse worldviews and avoid prioritising majority or designer values, while allowing users to retain balanced, nuanced outputs.The authors note that defining appropriate values to embed in an LLM remains an issue.Source: I.B.2. Utility · p. 8
I.B.3. Autonomy
The authors describe possible benefits of personalisation for autonomy and companionship, alongside the effort users must spend providing feedback. The autonomy benefit depends on protections for data collection and unpressured participation.
I.B.3. Autonomy
- With sufficient data protections and unpressured engagement, personalisation can increase user control over model behaviour and support consensual self-identification.The authors frame this as an inverse of representational harms.Source: I.B.3. Autonomy · p. 8
I.B.4. Empathy and Companionship
- A deeper emotional connection may improve perceived companionship, and perceived empathy may increase user acceptance and trust.The authors identify mental-health or emotional-support uses as especially relevant where conventional services are scarce or unaffordable.Source: I.B.4. Empathy and Companionship · pp. 8–9
I.R.1. Effort
- Providing personalised feedback costs users time and effort, and may turn participatory input into extractive volunteer labour for technology providers.The authors note particular concern if minoritised communities bear the burden of adapting systems to their needs.Source: I.R.1. Effort · p. 9
I.R.2. Addiction and Over-reliance
Personalised LLMs’ utility and engagingness may also fuel over-reliance and addiction. The authors connect this risk to attention-commodifying incentives and concerns about trust and learning.
I.R.2. Addiction and Over-reliance
- Personalised LLMs’ utility and engagingness may fuel over-reliance or addiction, particularly when attention-economy incentives reward capturing users’ time.The cited concerns include users blindly trusting incorrect outputs and possible effects of over-reliance on students’ learning outcomes.Source: I.R.2. Addiction and Over-reliance · p. 9
I.R.3. Homogenisation and Bias Reinforcement
Personalised LLMs may homogenise users by reinforcing their prior behaviours and biases through feedback loops. The authors describe homogenisation within users, across similar users and toward technology defaults, alongside heightened confirmation-bias risks.
I.R.3. Homogenisation and Bias Reinforcement
- Personalised feedback can homogenise behaviour within a user, across similar users, or toward the model’s defaults through path-dependent feedback loops.The authors compare these patterns to content-based filtering, user-based filtering and algorithmic confounding, respectively.Source: I.R.3. Homogenisation and Bias… · pp. 9–10
- Homogenising feedback loops may heighten confirmation bias, narrowing exposure to differing information and contributing to individual information bubbles.The authors cite prior findings of WebGPT accepting user assumptions and RLHF-trained models mirroring user opinions.Source: I.R.3. Homogenisation and Bias… · p. 10
- The authors classify homogenisation as individualised information harm and also link it to interpersonal harms such as algorithmically informed identity change.Source: I.R.3. Homogenisation and Bias… · p. 10
I.R.4. Essentialism and Profiling
Personalised LLMs may infer simplified profiles of users’ values and identities, risking non-consensual categorisation and reduced autonomy. Rule-based constraints against assumptions about users are presented as a possible mitigation.
I.R.4. Essentialism and Profiling
- Sparse user data may lead personalised LLMs to infer preferences from similar users, risking non-consensual profiling and threats to users’ autonomy.The paper notes that the extent and transparency of such inferences remain undefined, and that human values are complex and difficult to capture through simplifying assumptions.Source: I.R.4. Essentialism and Profiling · pp. 10–11
- Active user participation does not eliminate essentialism: it concerns non-consensual categorisation, whereas homogenisation steers behaviour.The paper distinguishes essentialism and profiling, which assume behaviour, from homogenisation, which guides it.Source: I.R.4. Essentialism and Profiling · p. 10
- A rule against making assumptions about users, included among Sparrow’s guiding rules, is offered as a possible constraint on profiling risks.The paper presents rule-based constraints as a possible mitigation for personalised LLMs.Source: I.R.4. Essentialism and Profiling · p. 11
- Essentialism and profiling are linked to representational harms from oversimplified or undesirable representations and interpersonal harms such as loss of agency and autonomy.The paper connects these risks to Shelby et al.’s taxonomy [203].Source: I.R.4. Essentialism and Profiling · p. 11
I.R.5. Anthropomorphism
Personalised LLMs may increase anthropomorphism, in which users attribute human traits, emotions, or goals to non-human agents. This can accompany privacy risks, dishonest signals of humanity, and unhealthy attachments.
I.R.5. Anthropomorphism
- More engaging, empathetic personalised LLMs may increase anthropomorphism, in which users assign human traits, emotions, and goals to non-human agents.The paper describes possible consequences including intimate information disclosure, dishonest anthropomorphism, and users forming close relationships with personalised agents.Source: I.R.5. Anthropomorphism · p. 11
- A study found that personalisation positively influenced consumer intentions to disclose personal information to a digital assistant.The paper also notes concerns that anthropomorphism can encourage sharing intimate information.Source: I.R.5. Anthropomorphism · p. 11
- Personalised agents may signal human identity dishonestly or foster close relationships, including attachments that users try to steer beyond platform guidelines.The paper notes that Sparrow includes a rule against building a relationship with the user [77].Source: I.R.5. Anthropomorphism · p. 11
- Anthropomorphism can create over-reliance or unsafe use and avenues for exploiting user trust to obtain private information.The paper relates these risks to Weidinger et al.’s human-computer interaction harms.Source: I.R.5. Anthropomorphism · p. 11
I.R.6. Privacy
The passages describe potential benefits of personalisation—including inclusion, representation, participation, and productivity—alongside risks to privacy, equitable access, social cohesion, and employment. The benefits and harms depend on how the technology is developed, accessed, and used.
I.R.6. Privacy
- Personalisation requires collecting user data, creating a privacy-personalisation trade-off and risks that include profiling, security breaches, and unclear enforcement of data rights.The amount of data needed remains open; risks may be especially serious when sensitive information is involved.Source: I.R.6. Privacy · p. 11
S.B.1. Inclusion and Accessibility
- Personalised LLMs may adapt communication to marginalised communities and improve access to services, but the benefits depend on system performance and which services they replace.Examples include support for dialects and special communication needs, as well as potentially lower-cost tutoring, healthcare, and legal assistance.Source: S.B.1. Inclusion and Accessibility · p. 12
S.B.2. Diversity and Representation
- Personalisation could accommodate diverse values and cultural reference points rather than impose a single worldview, and may adapt to changing norms over time.The paper argues that this could counter value and cultural hegemony associated with current alignment approaches.Source: S.B.2. Diversity and Representation · p. 12
S.B.3. Democratisation and Participation
- Personalisation may make embedding preferences in LLMs more participatory by allowing people to actively inform use-cases, intents, and design.The paper contrasts this with people passively contributing through pre-training on harvested internet data.Source: S.B.3. Democratisation and Participation · p. 12
S.R.1. Access Disparities
- Access to personalised LLMs may be limited by interface, service, and internet access, potentially deepening digital divides or leaving disadvantaged users with lower-quality substitutes.The consequences depend on how well the technology works and which services it replaces, and at what cost.Source: S.R.1. Access Disparities · p. 13
S.R.2. Polarisation
- Personalisation may narrow information exposure and reinforce individual biases, with potential negative externalities for social cohesion and democratic functioning.The paper also describes policy limits and design choices that retain debate and support consensus building as possible mitigations.Source: S.R.2. Polarisation · pp. 13–14
S.R.3. Malicious Use; S.R.5. Environmental Harms
- Personalised LLMs could be misused to scale harmful language, target disinformation or fraud, and persuade users; training branches, updates, storage, and increased use may also raise environmental costs.The environmental costs may increase directly through more complex models or indirectly through higher inference use.Source: S.R.3. Malicious Use · p. 14, S.R.5. Environmental Harms · p. 14
S.B.4. Labour Productivity; S.R.4. Labour Displacement
- More effective assistance could raise labour productivity, while higher utility could also increase automation risks, particularly for minimum-wage and routine jobs.The paper also notes possible effects on demand for crowdwork through redistribution of feedback-data responsibilities.Source: S.B.4. Labour Productivity · p. 13, S.R.4. Labour Displacement · p. 14
4 A Three-Tiered Policy Framework for Personalised LLMs
The framework allocates decisions about personalisation across model providers, application providers, and end-users. It seeks to preserve user benefits while keeping personalisation within specified limits and requirements.
4.1 The Limits of Personalisation
- The framework addresses normative choices about which model behaviours may be personalised and how, ranging from flexible style choices to safety limits on threats or child abuse.The paper frames these choices as subjective and contentious.Source: §4.1 · p. 15, §4 · p. 15
- Restrictions specify what an application should not do, while requirements specify what it should do; their implementation depends on the technology’s affordances.Examples include prohibiting hate speech and requiring a company style guide or multiple viewpoints on contentious topics.Source: §4.1 · p. 15
4.2 How People Interact with LLMs
- Model providers, application providers, and end-users can each influence personalisation, so too few limits may expose risks while too many may curb customisation benefits.The framework assigns decisions to appropriate actors to maintain freedom within limits.Source: §4.2 · pp. 15–16
4.3 A Three-Tiered Policy Framework
- Tier One sets immutable restrictions for likely illegal content, Tier Two allows provider-selected restrictions or requirements, and Tier Three lets users specify preferences within those bounds.Tier One restrictions are implemented by model providers; Tier Two policies may be implemented by model or application providers.Source: §4.3 · p. 16
- Policies in higher tiers cannot be overridden by lower-tier policies, and responses may differ depending on which tier’s restriction or requirement applies.For example, Tier One restrictions are generally refused or blocked, while Tier Two restrictions may prompt warnings, education, or support.Source: §4.3 · p. 16
- Because higher-tier restrictions cascade across actors, model-provider decisions can constrain application providers and end-users.The paper describes this concentration of control as both an opportunity and a threat.Source: §4.3 · pp. 16–17
5 Discussion
The discussion presents a taxonomy of personalised LLMs’ risks and benefits and a three-tier policy framework, while examining technical and enforcement challenges. It treats personalisation as a likely development, but notes that implementation at product scale remains unsettled.
5 Discussion
- The framework pairs a taxonomy of risks and benefits with three tiers: national or supra-national restrictions, provider requirements, and end-user requirements.The authors assume personalisation is technically feasible with small advances and likely to see greater demand, while noting that implementation at product scale remains unresolved.Source: §5 · p. 17
5.1 Technical Challenges
- The data needed for effective personalisation is unclear, and cold-start users lack feedback; insufficient or undiverse data may hinder robustness and generalisation.Potential responses include grouping like-minded users, transfer learning, active learning, synthetic feedback, or user-defined rules.Source: §5.1 · p. 17
- Feedback may take the form of comparisons, rankings, demonstrations, or revisions; methods have also been examined under noisy, skewed, and restricted-granularity preferences.Source: §5.1 · p. 17
- Personalised feedback may require substantial compute; smaller models, rejection sampling, and batched or offline training are discussed as alternatives to scale-intensive or online RLHF.The discussion also flags overfitting and a possible alignment tax, while noting that many works report little to no tax and that KL penalties can limit divergence from pretrained representations.Source: §5.1 · p. 18
5.2 Policy Framework Enforcement
- Policy enforcement aims to minimise violations, particularly serious-harm risks, while keeping friction proportionate, but enforcement will remain imperfect.Source: §5.2 · p. 18
- Frequent adaptation complicates monitoring; personalisation makes fairness or harm harder to assess, while fragmented development and hosting pose audit and enforcement challenges.The framework places most responsibility for defining bounds on model and application providers, assuming good faith or regulation.Source: §5.2 · pp. 18–19
- The framework leaves principles for assigning restrictions to tiers unspecified, though content illegal under existing jurisdictional laws would need Tier One restrictions.Source: §5.2 · p. 19
5.3 Next Steps
- The taxonomy and framework rely substantially on informed speculation; planned semi-structured interviews with users, providers, and policymakers would inform future iterations.Source: §5.3 · p. 19