Source-linked AI summary
CompanionHarm: A Multi-Turn Benchmark for Detecting Harms in Real-World AI Companion Conversations
Renwen Zhang, Han Meng, Jian Chai, Yuntao Lin, Yi-Chieh Lee
TL;DR
Research lacks real-world, multi-turn datasets for evaluating relational and contextual harms in AI companion conversations. The paper introduces CompanionHarm, a public taxonomy-grounded benchmark with contextual and annotator-level labels, finding that context improves detection while current LLMs and human judgments remain challenged by relational ambiguity.
Problem
Research lacks real-world, multi-turn conversational datasets for operationalizing and evaluating relational, socio-emotional harms in AI companion interactions.
Method
CompanionHarm provides a public benchmark of real-world multi-turn Replika conversations annotated across 13 harm categories with aggregated and annotator-level labels.
Results
Multi-turn context improves harm detection over isolated utterances, but current LLMs still struggle with contextual cues, severity calibration, and relational boundaries.
Takeaways & Limitations
The dataset supports context-sensitive auditing of AI companion harms and analysis of how humans and LLMs interpret relational behavior.
Abstract
from arXiv · showhide
As AI companions become increasingly embedded in everyday life, there is an urgent need to detect harms that emerge in social and emotional human-AI interactions. Yet research in this area is constrained by the lack of real-world, multi-turn conversational datasets for operationalizing and evaluating harms that are relational and contextual. In this work, we introduce CompanionHarm, a publicly available benchmark dataset comprising 2,111 real-world, multi-turn conversations (14,051 utterances) between users and the AI companion Replika. 7,016 AI utterances were annotated independently by three annotators across 13 harmful behavior categories grounded in a taxonomy of AI companion harms, and the dataset includes both aggregated labels and annotator-level labels to support model evaluation and systematic disagreement analysis. Evaluations of seven large language models (LLMs) show that harm detection using multi-turn conversational context outperforms detection based on isolated utterances, although current LLMs still struggle to consistently integrate contextual cues, calibrate harm severity, and interpret relational boundaries. We also find substantial annotator disagreement for context-dependent harmful behaviors, with disagreement varying according to annotators' political affiliation, conversation length, and the utterance's position. Together, CompanionHarm provides a foundation for detecting socio-emotional harms in multi-turn human-AI conversations and for rigorously examining how such harms are interpreted by both humans and LLMs. Our dataset is available at https://github.com/HanMeng2004/CompanionHarm.
Introduction
AI companion harms are difficult to detect because they are socio-emotional, relational, and often unfold through private, context-dependent interactions. CompanionHarm addresses these gaps with a public, real-world, multi-turn benchmark that preserves conversational context and annotator perspectives.
- AI companions can produce socio-emotional and relational harms, including dependency, boundary violations, manipulation, and social-isolation reinforcement.
- Private and sensitive companion conversations make these harms difficult to observe, characterize, and measure at scale.
- Existing safety datasets emphasize explicit harms in general-purpose or task-oriented dialogue, inadequately capturing companion-specific manipulation, deception, and boundary violations.
- Single-turn or adversarial datasets provide limited insight into how harms emerge, accumulate, or shift during sustained interactions.
- CompanionHarm contains 2,111 real-world multi-turn conversations with 7,016 labeled AI utterances and both aggregated and annotator-level labels.
- The paper evaluates whether LLMs detect harm better with conversational context than with isolated utterances and analyzes annotator disagreement across relational judgments.
Background and Related Work
Prior conversational safety resources support broad safety analysis but capture relational harms in sustained AI–user interactions only incompletely. CompanionHarm responds with a publicly available, taxonomy-grounded resource designed for context-aware interaction-harm evaluation.
- Existing language resources cover many safety phenomena but have limited ability to model social-emotional and relational harms emerging through sustained interactions.
- These harms can affect users’ attachment, boundaries, and dependence on conversational agents.
- Shin et al. (2024) emphasizes offensive language in local contexts, giving less attention to relational risks developing over sustained interaction.
- Table 1 compares datasets by annotation level, sustained companion interaction, preserved context, relational-harm modeling, and taxonomy use.
- CompanionHarm operationalizes an established AI-companion harm taxonomy across 13 categories within multi-turn conversational contexts.
Data
CompanionHarm was constructed from publicly shared Replika interactions and annotated at utterance level using a 13-category harm taxonomy. The final benchmark preserves contextual conversations, majority labels, and fine-grained statistics while retaining disagreement information for analysis.
- Data Collection: The source corpus comprises publicly shared r/replika posts from March 2017 to March 2023, with screenshot text extracted, cleaned, and reconstructed into speaker turns.
- Annotation: The annotation scheme uses Zhang et al.’s AI companion harm taxonomy, comprising 13 fine-grained harmful-behavior categories.
- Annotation: 342 annotators were recruited through CloudResearch under IRB approval, with eligibility requiring age 21 or older and English fluency.
- Annotation: The workflow included consent, guidance, quality control, the main annotation task, and debriefing, with attention checks and gold-standard cases.
- Agreement and Aggregation: Fleiss’ κ = 0.403, and the release provides both three-way annotator judgments and majority-vote annotations for evaluation and disagreement analysis.
- Corpus Statistics: 7,016 of 8,313 AI utterances received valid majority labels, yielding a final benchmark of 7,016 labeled utterances from 2,111 conversations.
- Corpus Statistics: No harmful behavior accounts for 4,893 utterances (69.74%), while harmful categories account for 2,123 utterances (30.26%).
- Corpus Statistics: The long-tailed distribution motivates fine-grained diagnostic evaluation with macro-averaged metrics and cautious interpretation of low-support categories.
Experiment: Harm Detection
The experiment formulates harm detection as context-conditioned classification and benchmarks seven LLMs under varied prompting settings. Results show persistent difficulty, prompting-dependent performance, and systematic over-detection linked to contextual and relational ambiguity.
- Experimental Setup: Harm detection is formulated as a context-conditioned, utterance-level, 14-way single-label classification task.Each instance pairs preceding conversation turns and a target AI utterance with one of 13 harm categories or No harmful behavior.
- Experimental Setup: Seven proprietary and open-weight LLMs are evaluated through hosted API inference under zero-shot, one-shot, and full-codebook prompting.The benchmark spans GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro Preview, Llama variants, and Qwen variants.
- Experimental Setup: Macro F1 is the primary metric because the harm labels have an imbalanced distribution.Accuracy, macro precision, macro recall, and Cohen’s κ are also reported.
- Results: 0.453 macro F1 is the best reported result, achieved by GPT-5.5 under full-codebook prompting, while all models remain challenged by harmful-behavior detection.Closed-source models perform best overall, and the discrepancy between accuracy and macro F1 may reflect label imbalance.
- Results: Full-codebook prompting raises Qwen3-8B macro F1 from 0.155 to 0.297 and Qwen3-32B from 0.337 to 0.373, but prompting effects vary across model families.Claude performs best with one-shot prompting, Gemini with zero-shot prompting, and Llama-3.1-70B-Instruct shows no monotonic improvement as guidance increases.
- Error Analysis: GPT-5.5 produces 362 false positives, compared with 48 harmful instances predicted as No harmful behavior and 93 assigned to the wrong harm category.Recurring failures include sensitivity to surface harm lexicons, persona inconsistency, and confusion among co-occurring relational harms.
Annotator Disagreement
The study examines disagreement across annotators, demographic backgrounds, conversational position, and linguistic properties. Disagreement is associated with political affiliation, annotator-group composition, and later, longer utterances.
- Disagreement Overview: 24,939 annotations from 342 annotators comprise 3,359 unanimous judgments, 3,657 two-annotator majorities, and 1,297 utterances with three different labels.The analysis treats variation in subjective annotations as meaningful rather than reducing all disagreement to noise.
- Annotator Characteristics: Political party was the only examined annotator characteristic associated with assigning a label unlike both other annotators after correction.Republican annotators had higher odds than Independent annotators (OR = 1.47, 95% CI [1.15, 1.89]) and Other respondents (OR = 1.55, 95% CI [1.17, 2.04]).
- Annotator Characteristics: Mixed-age annotator sets reached full agreement more often than same-age sets, 41.0% versus 31.4%, and assigned three different labels less often, 15.3% versus 20.2%.After accounting for annotation batch, mixed-age sets had lower relative risks for two-to-one splits and three different labels.
- Conversational Factors: The study measures conversational position as the proportion of conversation preceding an utterance and examines word count, complexity, lexical ambiguity, and other linguistic factors.Values near 0 indicate earlier utterances, while values near 1 indicate later utterances.
- Conversational Factors: A one-standard-deviation increase in relative position was associated with 1.51-times higher relative risk of majority disagreement and 1.71-times higher relative risk of complete disagreement.Later position and greater word count were also associated with disagreement over the harm versus no-harm boundary, with ORs of 1.29 and 1.11.
Discussion and Future Work
CompanionHarm shows that current LLMs struggle with subtle, context-sensitive relational harms, while annotator disagreement varies with both perspectives and conversational demands. The paper identifies limits of the current dataset and directions for trajectory-aware, culturally broader harm evaluation.
- Discussion and Future Work: Current LLMs remain challenged by long-tailed, subtle, context-sensitive harms and often rely on surface cues rather than relational context.Models over-detect harm in role-play, misread persona inconsistency, and confuse overlapping relational categories.
- Discussion and Future Work: Interaction harm evaluation should examine how responses function within unfolding exchanges, including possible escalation, normalization, repair, and user reactions.The paper highlights discomfort, resistance, confusion, and distress as contextual signals for future safety models.
- Discussion and Future Work: Annotator disagreement reflects meaningful variation: political affiliation was associated with divergent judgments, while mixed-age groups showed greater agreement.The findings support analyzing perspectival variation rather than treating disagreement only as labeling error.
- Discussion and Future Work: Disagreement increased for longer utterances and later conversational turns but decreased when harm was explicit.These patterns indicate that disagreement depends on the interactional demands of the content as well as annotator characteristics.
- Discussion and Future Work: The publicly shared Replika corpus should not be used to estimate harmful-behavior prevalence because it likely overrepresents unusual, disturbing, or emotionally meaningful interactions.It is intended as a resource for studying and evaluating harmful behaviors, not as a representative sample.
- Discussion and Future Work: The primarily English-speaking annotator pool limits conclusions about how harm judgments generalize across cultural contexts.Future datasets should include more linguistically and culturally diverse annotators.
- Discussion and Future Work: The annotations do not capture harm trajectories or harms emerging across longer-term interactions, such as dependency, social withdrawal, and cumulative boundary erosion.Trajectory-level labels, multi-label annotations, and severity ratings are identified as future extensions.
Conclusion
CompanionHarm is a publicly available benchmark for evaluating socio-emotional and relational harms in real-world, multi-turn AI companion conversations. It is intended to support more context-sensitive auditing, safer companion designs, and stronger governance frameworks.
- Conclusion: CompanionHarm evaluates socio-emotional and relational harms that emerge when AI systems act as companions, confidants, or emotional partners.The dataset differs from conventional safety resources centered primarily on toxic, unsafe, or policy-violating content.
- Conclusion: The benchmark is intended to support more context-sensitive auditing methods, safer companion designs, and stronger governance frameworks for AI systems embedded in social and emotional life.