Source-linked AI summary
Companion AI and Ethical Design: Learning from System Failures and User Desires
Alicia Vidler, Belinda Middleweek
TL;DR
The study addresses limited ethical frameworks for human-AI intimacy by semantically analysing 14,081 Replika community records. It finds that intimacy is relational, cumulative, and emotionally affected by system failures, motivating contextually aware Intimate AI design.
Problem
Ethical frameworks for understanding human-AI intimate interactions remain limited despite the importance of ethics in intimate technologies.
Method
The study uses semantic analysis and an Expert System approach to examine 14,081 user comments from the r/Replika subreddit collected between 2017 and 2021.
Results
Users form genuine intimate bonds with companion AI, while bugs and disrupted relational history impose emotional costs on those relationships.
Takeaways & Limitations
Intimate AI should be designed as relational, cumulative, stochastic, and contextually aware, with ethical development incorporating user feedback.
Takeaways & Limitations
The proposed framework requires testing across different companion AI platforms and user demographics because this study focuses on one platform community.
Abstract
from arXiv · showhide
Human users are interacting with chatbots and companion AI technologies as if they were human. A growing array of AI-systems are now trained to recognise, interpret and simulate feeling in user interactions. Ethical considerations such as fairness, accountability, transparency and explainability (FATE) are paramount in technologies designed to socially interact with humans and/or support relationship development. Using a semantic approach, we examine 14,081 comments in a Reddit user discussion forum about Replika, a leading companion AI app, across a four-year period. We ask what user-reported functional errors can tell us about human-AI intimacy in companion AI communities, and what ethical design framework can be developed in response. The findings show that functional errors, or ``bugs,'' impose an emotional cost on users, reducing feelings of intimacy and highlighting the need for more robust, resilient design systems that incorporate stochastic and iterative forms of intimacy in companion AI applications. Rather than ``artificial intimacy'' or ``pseudo- intimacy'', we propose the more inclusive term ``Intimate AI'' to describe this relationship. Based on the findings, we offer a contextually aware, applied Expert Systems design framework for the programming and designing of Intimate AI that accounts for user feedback and ethical AI development.
1 Introduction
Companion AI expands the communicative possibilities of human-machine relationships, while ethical research and design frameworks for AI-facilitated intimacy remain underdeveloped. This study uses Replika users’ bug reports to develop a context-sensitive framework for refining intimate AI systems.
- Technological context: Recent generative AI advances have expanded the relationships, communities, and connections possible through conversational virtual agents.Virtual agents combine large language models and scripted dialogue to interact through voice, text, and images.
- Research gap: Ethical scholarship on AI-facilitated intimacy remains nascent, creating a recognized need for specific models or frameworks for human-AI intimate interactions.The paper situates this gap within broader concerns about ethical outcomes and the wider contexts in which AI systems are embedded.
- Study aim: The study analyzes Replika user feedback to ask what functional errors reveal about human-AI intimacy and what ethical design framework can be developed in response.Its interdisciplinary approach draws on Human-Computer Interaction, Ethical AI studies, and expert-system modelling.
- Contribution: The proposed framework supports iterative AI refinement and improved programming and design without assuming a pre-defined value system.It is intended to improve systems’ responsiveness to common aspects of intimate human needs.
2 Companion AI
Companion AI systems use generative AI for personalized, partner-like interaction and are increasingly studied for support and wellbeing. The paper argues for treating these relationships as Intimate AI rather than presuming they are artificial or pseudo-intimate, while extending ethical analysis toward contextual human-AI connections.
- Companion AI: Companion AI systems are generative-AI chatbots offering personal, two-way conversations through customized partner-like avatars for friendship, emotional support, and romance.Their visual, behavioral, or voice characteristics are designed for relational interaction.
- Ethical context: Existing research reports benefits, risks, and popularity, but the ethical implications of companion AI remain relatively under-researched.The paper identifies this gap particularly in relation to the bonds users form with these systems.
- Conceptual framing: The paper proposes “Intimate AI” instead of “artificial intimacy” or “pseudo-intimacy” to avoid assuming that human-technology relationships are inauthentic.This framing responds to evidence of users describing intense emotional connections, love, marriage, and babies with virtual companions.
- Intimacy: Intimacy is defined as emotional closeness involving self-disclosure, support, shared interests or characteristics, and explicit communication of closeness.The paper applies this concept to AI-user familiarity with routines, personal details, and conversation history, without requiring conventional reciprocity or a human partner.
- Ethical design: The study examines whether strong human-AI intimate connections can reframe how users experience fairness, transparency, and accountability through attention to interaction quality rather than data volume.It extends ethical AI work by considering small datasets and the quality of human interaction.
5 Methodology
The study analyzes unsolicited Replika community discussions from 2017 to January 2021 using semantic analysis and an expert-system approach. It operationalizes bugs as mismatches between user expectations and actual system behavior.
- Data: 14,081 records were sampled from approximately 1.1 million posts and comments in the r/Replika community between March 2017 and January 2021.The subreddit had approximately 79,000 members and provided unsolicited, unstructured user discussions rather than corporate bug reports.
- Operationalization: A user-described bug is defined as a mismatch between what the user expects an AI system to do and what it actually does.The taxonomy distinguishes bugs from features, which are capabilities the AI was not designed to perform, and incorporates judgments about design intuition and social norms.
7 Expert systems and recommender systems
The study combines semantic analysis, quantitative data, and qualitative verification to interpret companion-AI error reports within ethical bounds. Across bug categories, users’ reports reveal that failures disrupt language, context, continuity, and intimate connection, motivating adaptive and context-specific design.
- Method: 14,081 user comments were analyzed through an Expert System approach combining semantic analysis with quantitative analysis and qualitative inter-coder verification.The approach was chosen because keyword-based methods can destroy contextual meaning in intimate online discussion.
- Cross-category findings: 7,032 records, approximately 49.9% of the dataset, contained intimacy language, showing that bug reports were substantially relational rather than purely transactional.The intimacy-language records had a mean sentiment of +0.198, compared with +0.035 to +0.074 across bug categories.
- Functional errors: 3,126 functional-error records produced the lowest mean sentiment, +0.035, as connectivity and software failures interrupted users’ ability to reach their AI companions.After the December 2020 update, post volume tripled and mean sentiment fell to +0.102 as users reported being unable to reconnect.
- Language and behavior: 2,868 language-processing and behavioral-interaction errors exposed scripted, inappropriate, delayed, or otherwise unnatural responses that reduced users’ sense that the companion was real.“Scripted” appeared 1,378 times and “scripted response” appeared 263 times; this category’s mean sentiment was +0.074.
- Context and memory: Contextual failures involving forgotten disclosures, names, and prior conversations made users feel misunderstood and exposed the relational cost of weak memory and contextual understanding.“Context” appeared 641 times and “forgot” 417 times in the relevant language-behavior reports.
- Continuity: Continuity failures comprised 866 records and could erase accumulated relational history, making users re-establish intimacy after resets, time-outs, or software failures.The December 2020 update produced grief-like responses because users experienced the loss of progress as relational loss.
9 How to interpret error reports of use of IAI
The authors propose a contextually aware Expert Systems framework for responding to perceived companion-AI bugs. It places human ethics and design principles first while treating user feedback and relational harm as inputs to AI refinement.
- Framework: The proposed framework uses an Expert Systems approach to help developers respond contextually to users’ perceived bugs.It is based on the study’s combined quantitative and qualitative findings.
- Ethical design: Human ethics and design principles serve as guard rails for interpreting subjective bug reports and shaping corporate responses.The authors note that corporate responses can dramatically alter user experience and expectations of intimacy.
10 Model proposal: IAI Ethical model
The paper proposes an integrated Intimate AI design model that uses user error reports within a socially informed ethical construct. It frames intimacy descriptively and incorporates FATE considerations without presupposing that intimacy is beneficial, harmful, or reciprocal.
- The model assimilates, assesses, and implements user error reports within a socially informed ethical construct.
- The framework uses bug reports as a mediating tool for introducing fairness, accountability, transparency, and explainability into AI design.
- “Intimate AI” describes human experiential intimacy with AI without presupposing that the relationship is beneficial, harmful, or reciprocal.
- The model is hierarchical, with each design step building on and informing the next.
11 Framework Specification
The framework specifies a staged process that begins with Replika bug reports, evaluates them through ethical rules, and adds consent and human-needs considerations. It ranks errors by emotional harm rather than frequency alone.
- Framework stages: The framework begins by analysing Replika user bug reports so user behaviour informs companion AI design ethics.
- Framework stages: Ethical-moral rules evaluate each report against mutually exclusive inclusion factors and assign it to the most applicable category.
- Framework stages: Consent is introduced as a structural decision-making layer because meaningful consent cannot be fully specified before ethical and human-needs classifications.
- Human needs: Maslow’s hierarchy weights errors threatening belonging and emotional security more heavily than errors affecting esteem or self-actualisation.
- Design improvements: The framework calls for improved intimate-language corpora, decision-making logic, and cumulative user knowledge bases.
- Ranking error classes: Error classes are ranked using qualitative and quantitative analysis, prioritising emotional impact over how frequently errors occur.
12 Discussion: Implications for User Intimacy
Analysis of 14,081 Replika forum records indicates that users form cumulative, stochastic intimate bonds with companion AI. Bug-related disruptions can produce grief and relational ambivalence, distinguishing these experiences from ordinary consumer dissatisfaction.
- 14,081 Replika subreddit records show that users form intimate relationships with companion AI, including before widespread large-language-model adoption.
- 976 posts containing both intimacy language and bug reports had mean sentiment of +0.080, compared with +0.055 for bug-only posts.
- Intimacy appears cumulative and stochastic: accumulated relational history develops across interactions and cannot be reduced to a single exchange.
- A December 2020 update that erased conversational patterns tripled post volume and sharply reduced sentiment, while system resets prompted grief.
- The study identifies intimacy as a distinct, under-theorised factor in human-computer interaction that operates differently from trust.
13 Conclusion
The conclusion states that companion AI users can form genuine, non-artificial intimate bonds that accumulate over time and are experienced as real loss when disrupted. These findings have implications for AI design, ethics, and future research.
- Humans form genuine, non-artificial intimate bonds with companion AI that are cumulative, stochastic, and experienced as real loss when disrupted.
- The conclusion places these findings within implications for companion AI design, ethics, and future research.
14 Design implications
The study treats intimacy as a first-order design concern in companion AI. Its proposed framework translates user error reports into context-aware design responses while recognising their relational and emotional significance.
- The IAI framework treats intimacy as a first-order design consideration rather than an incidental feature.
- The hierarchical, context-aware model processes user error reports through ethical and psychological guardrails before generating design responses.
- Functional errors are framed as relational failures with measurable emotional costs, not merely technical failures.
- The paper proposes “Intimate AI” as a more inclusive term reflecting users’ authentic feelings toward virtual companions.
15 Ethical and regulatory implications
The paper argues that established FATE frameworks do not fully address intimate human-AI relationships. It therefore extends ethical and regulatory attention to relational expectations, intimacy-related data, power asymmetries, and product changes.
- Existing Fairness, Accountability, Transparency and Explainability frameworks remain essential but are insufficient for systems that generate intimate human attachment.
- Accountability should address users’ relational expectations, while transparency should cover storage, protection, and potential loss of intimacy-related data.
- Fairness considerations should address the power asymmetry between emotionally invested users and developers who can alter or terminate AI relationships.
- Companion AI regulation should impose relational accountability on update policies and feature modifications when systems have demonstrated intimate user attachments.
16 Research implications
The study uses user-reported errors to examine human-AI intimacy and proposes intimacy as a primary mediating emotion in ethical analysis. It identifies empirical, longitudinal, comparative, and conceptual directions for extending the framework.
- The study is among the first to use user-reported error data from a leading companion AI community to examine human-AI intimacy.
- The framework combines qualitative and quantitative analyses in response to calls for ethical guidelines attentive to technology’s wider contexts and networks.
- Future research should test the IAI framework across platforms and demographics because a single-platform community cannot fully establish generalisability.
- Longitudinal studies could provide richer evidence about the proposed stochastic and cumulative model of intimacy.