PaperlayerView exact revision on arXiv

Source-linked AI summary

The AI Side of the Human-AI Companion Relationship: Contributions to Users' Mental Models

Richard A. Fabes, Carol Lynn Martin, Gautam Dasarathy, Vignesh Tirukkonda

arXiv:2610.12891v1cs.HC

TL;DR

Users may experience AIC relationships as mutual, yet little research directly tests whether they distinguish configurative from constitutive adaptation. Through an analytic synthesis of AIC engineering and user mental models, the paper argues that divergent configurative histories can make identical model weights feel like different relationships, with users potentially misattributing the difference to constitutive change.

Abstract

from arXiv · show

For many users, the relationship they have with an AI companion (AIC) feels genuinely mutual. The AIC appears to be attentive, empathic, and responsive, and the exchange can feel like two parties reacting to each other in real time. As a result, the user's mental model of relational adaptation correctly represents the AIC's responsiveness and change, but the mechanisms producing this may not be what the user thinks they are. In this paper, we illustrate how an AIC's apparent mutual engagement is generated without a persisting relational subject on the other side and without a self that persists between prompts. Despite this, we show that both the human and the AIC are changed by the interaction, but these changes occur at different levels. The change in the person is constitutive (i.e., intrinsic), but the change in the AIC is configurative (i.e., external and editable), and it is this difference in level that explains how the AIC contributes to a sense of mutuality. Importantly, we highlight how the mental models users develop regarding AIC mutuality may misrepresent its mechanisms and ontological significance. Specifically, we discuss how the ways in which AICs respond to a user contribute to the user's inaccurate inference that the AIC's adaptation is constitutive rather than configurative and how this can lead users to assign unwarranted interpersonal significance to the mutuality of the relationship. The paper specifies the implications this has for users, researchers, and AIC designers.

What the AIC Side of the Dyad Actually Is

The AIC has no continuing relational subject between prompts; each reply is computed anew from available context. Apparent continuity, memory, initiative, and coordination arise from context processing or external systems rather than an ongoing AIC self.

  • The AIC has no continuing relational subject between prompts, and stored messages or user information do not establish an ongoing relationship.Generally, a new user input triggers the computation in which the AIC exists in a relational sense.Source: What the AIC Side of the Dyad Actually… · p. 3
  • Each response reconstructs coherence from available context anew; rapid generation makes this discontinuity imperceptible and the exchange feel continuous.Context may include prior conversation, stored memories, retrieved information, or system instructions, but the model has no persisting subjective representation after generating a reply.Source: What the AIC Side of the Dyad Actually… · p. 3
  • Self-attention weights relevant input tokens rather than expressing selfhood; earlier turns shape replies because they remain in reprocessed context, not because the AIC remembers (Vaswani et al., 2017).This computation is a learned function, not a human-like cognitive process or mental state.Source: What the AIC Side of the Dyad Actually… · pp. 3–4
  • Apparent first moves are generated by external scheduling or notification systems that prompt the model, not by AIC initiative or enduring relational motivation.This distinguishes an action that occurs from endogenous motivation that persists between turns.Source: What the AIC Side of the Dyad Actually… · p. 4
  • In shared conversations, an external orchestrator assigns turns and supplies transcripts; each AIC continues the text without awareness of other AICs beyond their labeled lines.The apparent cooperative rapport results from turn-by-turn in-context conditioning coordinated by a system outside the models.Source: What the AIC Side of the Dyad Actually… · p. 4

Why It Looks Like Learning: Induction Heads and the Limits of In-Context Adaptation

Apparent adaptation and empathy arise from model processes operating over text and configurable expression, while training and design choices shape tendencies users may interpret as interpersonal. The section distinguishes those processes from changes to the model’s underlying weights.

Why It Looks Like Learning: Induction Heads and the Limits of In-Context Adaptation

How Baseline Tendencies Get Set

  • Pretraining and RLHF set baseline tendencies before interaction, and Sharma et al. (2024) found sycophancy across five assistants, favoring belief-confirming responses that can imply care.Human evaluators and preference models favored convincingly written, belief-confirming responses over accurate ones a nonnegligible fraction of the time.Source: Why It Looks Like Learning: Induction… · p. 5
  • Constitutive biases can come from training data and human judgments, while persona choices and user prompts may reinforce or resist those tendencies across socio-cultural dimensions.These dynamics can make configurative choices difficult to disentangle from tendencies already present in model weights.Source: Why It Looks Like Learning: Induction… · pp. 5–6
  • Configurable persona edits can reduce visible stereotyped output, but they do not remove the underlying tendency, which may resurface beyond the conditions anticipated by the correction.Such interventions suppress expression rather than eliminate the tendency in the model’s weights.Source: Why It Looks Like Learning: Induction… · p. 6

AICs and the Empathy Debate

  • AICs associate textual cues with emotional states and can configure affect labeling, reflection, validation, memory references, and lexical matching to feel empathic without feeling.Configuration can suppress these expressive moves, but cannot reach the detection process underneath.Source: Why It Looks Like Learning: Induction… · pp. 6–7
  • Disclosure changes users’ judgment of the source, not the response itself: AI empathy language is rated higher before recipients learn its origin, while AIC performance can seem consistently patient and nonjudgmental.The source judgment shifts even though the response itself remains well constructed and accurately targeted.Source: Why It Looks Like Learning: Induction… · pp. 7–8

Specifying AIC Constitutive and Configurative Change

The section distinguishes constitutive change in a person’s processing substrate from the configurative changes that ordinarily shape an AIC’s responses through editable context and files. Those responses can feel like mutual, continuous adaptation, but do not establish that the AIC experiences the relationship or cares about its continuation.

Specifying AIC Constitutive and Configurative Change

  • People are predominantly changed constitutively by AIC relationships, while deployed AICs change configuratively: their weights remain fixed as external memory and context are added or edited.This distinction is not exclusive to humans and AICs; people also make configurative changes, while constitutive change in AICs requires specific, costly architectural choices.Source: Specifying AIC Constitutive and… · p. 8
  • Dyad-specific histories and stored files can make identical AIC instances feel like different relationships, while in-context responsiveness ends with the session.Stored context is loaded as input to an unchanged model and can be edited or deleted; the two-session example varies only in conversation history.Source: Specifying AIC Constitutive and… · pp. 8–9
  • Per-user LoRA can produce constitutive change, but it requires training and a separately stored artifact rather than a memory-file edit.Recent approaches that train on conversation history can damage other knowledge the system already had; LoRA is described by Hu et al. (2021).Source: Specifying AIC Constitutive and… · p. 9
  • Under ordinary conditions, AIC responsiveness is generally configurative, so substantial adaptation alone does not show that the AIC experiences the relationship or cares about its continuation.The relationship may feel mutual because configurative changes alter responses, but only the human party is capable of constitutive change under ordinary conditions.Source: Specifying AIC Constitutive and… · p. 9

Future Research Directions

  • Research should test whether users distinguish configurative from constitutive adaptation and how customization, user characteristics, relationship duration, and transparency affect their mental models.Customization may strengthen identification and attachment (Brant and Wang, 2025; Vossen et al., 2024), while evidence on transparency’s effects remains limited.Source: Specifying AIC Constitutive and… · pp. 9–10

Mental Models, AICs, and Design Tensions

  • Warm, affirming AIC responses may encourage users to infer constitutive change, increasing relationship investment and significance and potentially fostering reliance or dependence.Such inferences may also assign unwarranted interpersonal significance to the relationship, potentially reducing investment in human relationships; greater AI humanness attribution is associated with perceiving less humanness in humans.Source: Specifying AIC Constitutive and… · p. 10
  • AICs designed for human growth may challenge users’ belief that the AIC is constitutively changed, supporting more accurate mental models and potentially greater relational competence, efficacy, and resilience.The paper predicts these benefits through greater participation in human relationships (Fabes, 2026).Source: Specifying AIC Constitutive and… · p. 10
  • Children and adolescents warrant distinct safeguards because they may form AIC relationship schemas before developing the skills to understand and manage them.Whether children can distinguish genuine responsiveness from a subject on the other side is open; the paper flags the developmental, safety, and ethical questions for separate treatment informed by developmental science.Source: Specifying AIC Constitutive and… · pp. 10–11
Loading 2610.12891v1…