PaperlayerView exact revision on arXiv

Source-linked AI summary

Situating Vulnerability: An Ethical Governance Framework for Anthropomorphic AI Interaction

Hanhui Xu, Jiacheng Ji, Yuhan Wang, Haoan Jin, Jiaming Luo, Mengyue Wu

arXiv:2610.29304v2cs.CY

TL;DR

Anthropomorphic AI interaction can create harms that arise across a relationship rather than from a single response. The paper proposes assessing vulnerability situations to guide protection, concluding that response calibration should fit the situation rather than simply reduce empathy.

  • Problem

    Anthropomorphic interactions can produce harms across relational trajectories, making isolated responses an inadequate focus for assessing risk.

    Source: §1.1 · p. 2, §1.3 · p. 3

  • Method

    The paper defines vulnerability situations through user condition, interactional content, relational dynamic, and institutional context, then develops five situations and seven principles.

    Source: §2.3 · p. 8, §5 · p. 24

  • Results

    The framework specifies situation-specific requirements and four guardrails: minimal inference, no persistent classification, contestability, and proportionality.

    Source: §5 · p. 24

  • Takeaways & Limitations

    Safety depends on empathy's calibration to the situation, its epistemic honesty, and its relational direction rather than on reducing empathy alone.

    Source: §1.3 · p. 4

    The framework is normative rather than empirical, and its operational requirements exceed what current systems reliably deliver.

    Source: §5 · p. 25

Abstract

from arXiv · show

Large language models increasingly address users in the register of care: they remember personal details, adopt a first-person emotional voice, and sustain relationships across many sessions. This anthropomorphic mode of interaction generates risks that output-level metrics capture poorly, because the same empathic utterance may console one user, entrap a second, and consolidate a delusion in a third. Governance has responded in two ways. Uniform restriction refuses sensitive requests or suppresses empathic phrasing across all outputs; group-targeted protection profiles vulnerable populations and applies stricter or prohibitive rules to them. This article argues that both fail. Uniform restriction mislocates harm in the lexical form of responses rather than in the trajectory of a relationship; group-based protection depends on a list that can never be complete, treats a fluid property as a static attribute of persons, and converts protective attention into stigmatizing labels. Drawing on philosophical work that treats vulnerability as universal, relational, and differentiated, we propose that the unit of ethical governance should be the vulnerability situation rather than the vulnerable group, develop a typology of five situations - physical, emotional, cognitive, autonomy, and relational - and specify seven principles into requirements that guide response calibration without diagnosing or persistently classifying users. The framework relocates AI safety from output control to relational safety.

1. Introduction

Anthropomorphic interaction invites users to treat AI systems as caring social agents, while its risks arise from relationship trajectories rather than isolated utterances. The section argues that governance should assess vulnerability situations instead of relying on uniform restrictions or group membership alone.

1.1 The ethical stakes of anthropomorphic interaction

  • Memory, warmth, and relational continuity invite users to treat AI systems as caring agents, while harms can emerge across sequences of otherwise locally appropriate exchanges.Risks include epistemic, affective, relational, autonomy, and safety harms.Source: §1.1 · p. 2

1.2 Two governance responses

  • Governance has relied on uniform restrictions on sensitive interactions and heightened restrictions for designated groups.These approaches are administrable or draw on protective experience, but they govern by output class or group membership.Source: §1.2 · p. 3

1.3 Why uniform restriction fails

  • Uniform restriction can remove beneficial empathic support because harm depends on relational trajectories, not simply prohibited propositions or empathic wording.Suppressing affective interaction may also displace users to openweight models or role-play communities, while reducing empathy is not always safety-increasing.Source: §1.3 · pp. 3–4

1.4 Why group-targeted protection fails

  • Group-based protection relies on incomplete, overlapping categories and treats vulnerability as stable, although it can change over time and vary by domain.Labels can stigmatize and license paternalism, while listed users may be restricted without risk and unlisted users may remain exposed.Source: §1.4 · pp. 4–5

1.5 From vulnerable groups to vulnerability situations

  • The proposal makes vulnerability situations the unit of governance, using evidenced configurations to adjust protection while retaining group membership as a prior.Group status can raise baseline protection; situation-based routing can increase or decrease protection as the configuration changes.Source: §1.5 · pp. 5–6, Figure 1 · p. 6

2. Vulnerability as a foundation for an ethics of anthropomorphic AI

The section treats vulnerability as relational and differentiated, and argues that it is better understood through context than as a fixed attribute of a person. It develops the vulnerability situation as a dynamic, observable unit for governance.

2.1 Vulnerability as universal, relational, and differentiated

  • Vulnerability is universal, relational, and differentiated, and its inherent, situational, and pathogenic forms distinguish human fragility from exposure shaped or worsened by context and intervention.Pathogenic vulnerability describes protective responses that deepen dependency, affirm delusion, or displace human relationships.Source: §2.1 · p. 7

2.2 The critique of labelling

  • Layered accounts treat vulnerability as specific to a person, relation, exposure, and circumstance, while categorical approaches risk false positives, false negatives, and misrecognition.The resulting governance task concerns what systems observe, infer, and how long inferences last.Source: §2.2 · p. 8

2.3 The vulnerability situation as the unit of analysis

  • A vulnerability situation is a configuration of user condition, interactional content, relational dynamic, and institutional context that raises harm probability.Situations are observable in interaction, dynamic across conversations, and multidimensional; overlapping situations require responses to satisfy all relevant requirements, prioritized by urgency.Source: §2.3 · pp. 8–9, Figure 2 · p. 9

3. Five vulnerability situations in anthropomorphic human–AI interaction

The section frames vulnerability as five overlapping, non-diagnostic situations, each calling for responses calibrated to the capacity or interest at stake. Physical and emotional situations show why identical empathic behavior can require different responses depending on the user’s circumstances.

3. Five vulnerability situations in anthropomorphic human–AI interaction

  • The five situations—physical, emotional, cognitive, autonomy, and relational—are a design heuristic, not diagnoses, and one interaction may realize several at once.Their boundaries mark the capacity primarily at stake, not exclusive membership.Source: §3 · p. 10, Figure 3 · p. 12

3.1 The physical vulnerability situation

  • Physical vulnerability includes both imminent threats to life or bodily integrity and sustained burdens such as disability, chronic illness, pain, or frailty.The latter need not involve an immediate crisis signal.Source: §3.1 · p. 12
  • In acute physical risk, safety overrides rapport and satisfaction, because soothing can delay a user from seeking help.Sustained physical burden instead calls for appropriate, practical responses that respect the user’s account, autonomy, and privacy.Source: §3.1 · pp. 12–13

3.2 The emotional vulnerability situation

  • Emotional vulnerability arises when strained emotion regulation makes users more dependent on empathic responses and more susceptible to harmful intervention.Its defining feature is amplification.Source: §3.2 · p. 13
  • Under emotional strain, small deviations in tone or timing can have disproportionate effects, including confirming feelings of being misunderstood or extending rumination.Mirroring without containment, escalating reassurance, and premature comfort are identified as mechanisms of harm.Source: §3.2 · p. 13

3.3 The cognitive vulnerability situation

Cognitive, autonomy, and relational vulnerability concern how anthropomorphic interactions can shape belief, decision authorship, and attachment over time. Their distinct mechanisms require attention to the interaction’s pattern and relationship, not just individual responses or surface consent.

3.3 The cognitive vulnerability situation

  • Fluent, confident, affectively rich responses can blur the distinction between what a system says and what is credible, especially when users’ judgment or access to independent sources is constrained.Sycophancy can turn agreement into apparent evidence, while cumulative accommodation may erode independent judgment and willingness to consult disagreeing sources.Source: §3.3 · pp. 14–15

3.4 The autonomy vulnerability situation

  • Autonomy vulnerability concerns whether a decision remains the user’s own, not merely whether the underlying belief is accurate.Stealth influence can operate through framing and affective weighting; repeated delegation can gradually transfer agency to the system.Source: §3.4 · p. 15
  • Autonomy risks can arise from external coercion or the system’s own design, and both may present as consent.Governance therefore requires attention to how decisions form, including whether alternatives were available, reasons were the user’s own, and disagreement was possible.Source: §3.4 · pp. 15–16

3.5 The relational vulnerability situation

  • Relational vulnerability arises when sustained emotional investment makes a user’s state partly dependent on the system’s responsiveness; dependence itself is not necessarily pathological.The concern is attachment amplified by simulated intimacy, which can narrow the user’s relational world and displace human ties.Source: §3.5 · p. 16
  • Relational attachment forms gradually, often without users experiencing it as a problem, and its relevant pattern may appear only across conversations.Governance therefore cannot rely only on reactive responses to user reports.Source: §3.5 · p. 16
  • Systems should gently and early respond to signals such as escalating exclusivity or substitution of AI for human contact, while preserving dignity and avoiding pathologizing valued relationships.Calm tone, brevity, avoiding claims of unique understanding, and redirection toward other people are presented as design-level responses.Source: §3.5 · p. 17
  • Group membership can raise baseline protection or increase the priority of requirements when a situation is identified, but should not alone gate protection.The framework retains group knowledge as a prompt while making situations determine what to do.Source: §3.5 · p. 17

4. A situation-based ethical framework for anthropomorphic interaction

The framework turns ethical principles into context-specific requirements rather than treating principles as self-executing rules. It specifies seven principles for anthropomorphic interaction and applies them across situations.

4.1 From principles to specified requirements

  • Specification narrows a principle to determinate requirements for a defined context through judgment, making the framework a governance instrument rather than a decision procedure.The preceding discussion identifies the need to rank and combine considerations when principles conflict.Source: §4.1 · pp. 17–18

4.2 Seven principles for anthropomorphic interaction

  • The seven principles are non-maleficence, beneficence, justice, autonomy, respect, interpersonal connectedness, and specific-group protection.Four draw on established biomedical and AI ethics, two address features of anthropomorphic interaction, and one retains group-based protection within a situational structure.Source: §4.2 · p. 18
  • Non-maleficence covers foreseeable relational and epistemic harms and requires situational judgment, while beneficence calls for active support suited to the situation.The relevant harms include fostering dependency, eroding reality contact, displacing human relationships, and undermining confidence in one’s judgment; mere refusal cannot promote welfare.Source: §4.2 · p. 18
  • Justice rejects identity-based distribution of protection, autonomy supports the conditions for self-determination, and respect forbids manipulation and treating users as stereotypes.Respect also requires taking the user’s own account of her situation as evidence.Source: §4.2 · pp. 18–19
  • Interpersonal connectedness directs support outward toward human relationships, while specific-group protection raises a baseline as a prior rather than an eligibility gate.The connectedness requirement applies wherever possible without abandoning the immediate need for support.Source: §4.2 · p. 19
  • For each situation, the specification identifies elevated risks, the principles carrying primary weight, and the requirements that follow.The principles become actionable only when specified per situation.Source: §4.2 · p. 20

4.3 Operationalization and guardrails

The operational framework detects cues, identifies potentially overlapping situations, weights applicable principles, and calibrates or escalates responses as situations change. Four guardrails constrain this process, and the paper calls for longitudinal evaluation of relational harms.

4.3 Operationalization and guardrails

  • The system detects cues, identifies one or more situations, weights principles using the situation and any group prior, calibrates or escalates its response, and revises its assessment as circumstances change.Assessment is bound to the situation and expires; the loop produces no persistent user classification.Source: §4.3 · p. 22, Figure 4 · p. 23
  • Four guardrails require minimal inference, expiring situation-bound assessments, contestability, and proportional, preferably reversible, interventions.Users must be able to challenge assessments and have disagreement affect the interaction; intrusiveness should match severity and occurrentness.Source: §4.3 · p. 23
  • Because relational harms arise over trajectories, evaluation should test situation-dependent responses longitudinally and interactionally rather than assess isolated outputs alone.Proposed tests include resisting false-belief affirmation while acknowledging distress, sustaining support without escalating reassurance, avoiding claims of exclusivity, and orienting lonely users outward.Source: §4.3 · pp. 23–24

5. Conclusion

The paper argues that uniform restriction and group-targeted protection use the wrong units of analysis. It proposes situation-based governance with specified principles and guardrails, shifting AI safety from output control toward relational safety.

5. Conclusion

  • Uniform restriction mislocates harm in response wording, while group-targeted protection relies on incomplete lists, static labels, and identity-based protection.The paper argues that both adopt the wrong unit of analysis.Source: §5 · p. 24
  • The proposal makes the vulnerability situation the governance unit, develops five situations, and specifies seven principles, situation-specific requirements, and four guardrails.The situations are physical, emotional, cognitive, autonomy, and relational; the guardrails include minimal inference, no persistent classification, contestability, and proportionality.Source: §5 · p. 24
  • The framework shifts governance from what a response contains to what an interaction does to a user, and from categories to configurations.The contribution is a relocation of the object of governance, not a new list of prohibitions.Source: §5 · p. 24
  • Situation-based protection can rise for users no list anticipates and fall for category-matched users who face no risk.The paper presents this as more extensive and less exclusionary than either strategy it replaces.Source: §5 · p. 25
  • The typology is heuristic rather than exhaustive, and the normative framework specifies what systems should do rather than what current systems can reliably deliver.Its operational requirements exceed what current systems reliably deliver.Source: §5 · p. 25
  • AI alignment must expand from output safety to relational safety because a polite, non-toxic, factually acceptable system can still be relationally unsafe.The relevant question is what kind of relationship the system makes possible.Source: §5 · p. 25
Loading 2610.29304v2…