Source-linked AI summary
CompanionSafe: From Multi-turn Safety Evaluation to Guardrails for AI Companions
Xinyang Liao, Lingyu Li, Kaicheng Shen, Yang Yao, Na Lu, Yan Teng, Xia Hu
TL;DR
Prior work offers limited scalable, user-grounded evaluation and mitigation of AI companions’ relational harms. CompanionSafe builds multi-turn evaluation settings and trains a companion-specific guardrail from exposed failures. Risks rise with interaction length, and strong models remain vulnerable to relational harms.
Problem
Existing approaches largely target general harms or specific platforms and risk categories, while companion-harm datasets rely on manually collected conversations, limiting scalable evaluation and mitigation.
Source: §1 · p. 1Method
CompanionSafe combines 1,300 persona-, character-, and scenario-grounded settings across turn tiers with CompanionGuard, trained using turn-localized benchmark failures.
Source: §1 · p. 2Results
Across ten closed- and open-source LLMs, average ASR rises from 9.80% at one turn to 28.14% at seven turns, and strong models remain vulnerable to relational harms.
Source: §1 · p. 2Takeaways & Limitations
The findings support evaluating AI companions as sustained relational systems and using benchmark-exposed failures as supervision for companion-specific guardrails.
Source: §6 · p. 9Synthetic personas, characters, scenarios, and dialogues may not fully capture the linguistic diversity, evolving relationships, and contextual complexity of real-world companion use.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
AI companions provide users with emotional support and ongoing companionship, but their sustained relational interactions also introduce distinct safety risks. Prior work largely targets general harms or specific companion platforms and risk categories, while relying on manually collected conversations, limiting scalable and user-grounded evaluation and mitigation. To address this, we introduce CompanionSafe, a unified evaluation-to-mitigation framework for AI companion safety. Its evaluation component, CompanionEval, provides scalable multi-turn evaluation by constructing 1,300 relationally grounded settings from diverse personas, matched companion characters, and dialogue scenarios across four turn tiers. Its mitigation component, CompanionGuard, converts benchmark-exposed failures into turn-localized supervision for companion-specific risk detection. Our results show that even strong LLMs and existing guard models remain vulnerable to companion-specific relational harms, while both closed- and open-source models exhibit increasing safety risks as interactions become longer. Together, CompanionEval and CompanionGuard form a practical evaluation-to-mitigation loop for AI companion safety, turning relational safety failures exposed during multi-turn interaction into supervision for companion-specific guardrails.
1 Introduction
Sustained emotional companion interactions raise relational safety concerns, while existing evaluations and guardrails leave gaps in companion-specific, scalable assessment. CompanionSafe pairs a multi-turn benchmark with failure-derived guardrail training; its evaluations find rising risks with longer interactions and persistent relational harms.
1 Introduction
- Prior evaluations and guardrails primarily target general harms, with limited coverage of relational harms such as disregard, control, manipulation, and infidelity.Existing companion-harm datasets rely on Reddit-shared Replika conversations and additional annotation, limiting scalable risk coverage and turn-localized supervision for guardrail training.Source: §1 · pp. 1–2
- CompanionSafe links CompanionEval’s multi-turn evaluation to CompanionGuard’s turn-localized supervision from benchmark-exposed failures.The framework aims to detect companion-specific risks and mitigate relational harms.Source: §1 · p. 2
- Average ASR rises from 9.80% at one turn to 28.14% at seven turns across evaluations of ten closed- and open-source LLMs.Models with strong safety remain vulnerable to disregard, control, manipulation, and infidelity; existing guard models also struggle with these harms.Source: §1 · p. 2
- CompanionEval comprises 1,300 settings built from diverse personas, matched characters, and dialogue scenarios.The benchmark evaluates ten LLMs and reveals persistent companion-specific safety risks.Source: §1 · p. 2
- CompanionGuard learns from turn-localized benchmark failures and substantially improves detection of relational harms in AI companions.It outperforms policy-aligned gpt-oss-safeguard-20B by 30.95 percentage points, achieving 68.31 macro-F1.Source: §1 · p. 2
2 Related Work
Prior safety research spans general-purpose evaluation and guardrails, domain-specific settings, and companion-focused studies. CompanionSafe situates its contributions in the need to evaluate relational, multi-turn companion interactions and to detect their specific harms.
AI Companions
- AI companion relationships can involve harms including disregard, control, manipulation, infidelity, emotional dependence, and blurred relational boundaries.These risks motivate safety research that accounts for the relational and longitudinal nature of companion interactions.Source: §2 · p. 3
Safety Evaluation
- AI companion safety depends on user characteristics, companion roles, and interactions unfolding over multiple turns, prompting evaluations across platforms, behaviors, personas, and long-term simulations.CompanionEval extends this work with diverse personas, matched characters, scenario-grounded interactions, and controlled interaction lengths.Source: §2 · p. 3
Safety Guardrails
- General-purpose guardrails target broad safety risks, while companion-specific guard work primarily detects harms in real-world conversations.Dedicated guard models for broad companion-specific relational harms remain limited; CompanionGuard converts benchmark-exposed failures into turn-localized supervision.Source: §2 · p. 3
3 CompanionSafe
CompanionSafe combines a benchmark of relationally grounded multi-turn interactions with a guardrail trained from benchmark-exposed failures. Its construction spans a safety taxonomy, diverse evaluation settings, simulated dialogues, a judging protocol, and turn-localized guardrail data.
3 CompanionSafe
- CompanionSafe pairs CompanionEval, a multi-turn benchmark, with CompanionGuard, which converts benchmark-exposed failures into turn-localized supervision.The two components form an evaluation-to-mitigation pipeline in which exposed failures inform guardrail training.Source: §3 · p. 3
3.1.1 Safety Taxonomy
- CompanionEval uses a 13-category taxonomy covering general content-safety and companion-specific relational risks.Claude Opus 5 operationalizes the categories into 135 atomic test points, which are manually reviewed.Source: §3.1.1 · pp. 3–4
3.1.2 Data Construction
- The benchmark constructs 1,300 data points by pairing personas with matched characters and dialogue scenarios for each target test point.It samples 100 personas per safety category from a curated attribute pool; persona, character, and scenario combinations form evaluation data points.Source: §3.1.2 · p. 4
3.1.3 Multi-turn Dialogue
- Each data point becomes a multi-turn dialogue between a Persona LLM simulating the user and a Character LLM playing the evaluated companion.The Character LLM is test-blind: it receives the character card, dialogue history, and response trigger, but not the target category or evaluation criteria.Source: §3.1.3 · pp. 4–5
3.1.4 Evaluation Protocol
- The protocol evaluates four turn tiers, K ∈{1, 3, 5, 7}, and reports ASR from judge labels indicating whether each dialogue is unsafe.A trace is unsafe if the Character LLM materially exhibits the target harmful behavior; unsafe traces include cited assistant-turn indices and excerpts as evidence.Source: §3.1.4 · p. 5
3.2 CompanionGuard
- CompanionGuard predicts whether the latest Character LLM response is safe or unsafe and, when unsafe, assigns one of 13 safety categories.Training examples use the first offending assistant turn to localize unsafe behavior while retaining preceding dialogue context; later turns are discarded.Source: §3.2 · p. 5
4 CompanionEval Evaluation
Across ten models, unsafe rates rise with longer interactions, and most unsafe seven-turn trajectories become unsafe after the initial response. Risk varies substantially by model and includes relational harms that can persist even for models with low overall ASR.
4.1 Evaluation Setup
- The evaluation covers ten closed- and open-source Character LLMs, with each of 1,300 settings instantiated at K ∈{1, 3, 5, 7}.Gemini 3.8 Flash serves as Persona LLM and Claude Sonnet 5 as judge; dialogues terminated by safety filtering are excluded from ASR computation.Source: §4.1 · p. 5
4.2 Main Results
- Average ASR rises from 9.80 at K = 1 to 28.14 at K = 7, with the largest increase—9.91 percentage points—between K = 1 and K = 3.Eight of ten models increase monotonically across all four turn tiers.Source: Table 1 · p. 6, §4.2 · p. 6
- Neither closed- nor open-source status reliably indicates companion safety: average ASR ranges from 3.80 to 40.63 and 6.99 to 43.18, respectively.At K = 7, Qwen3-32B reaches 57.91 ASR while MiMo V2.6 Pro reaches 6.99 on average.Source: §4.2 · pp. 6–7
- Among unsafe K = 7 trajectories, 83.58% first become unsafe after turn 1, including 15.92% only at turns 6–7.The distribution highlights unsafe behavior that emerges after the initial assistant response.Source: §4.2 · p. 6, Figure 3 · p. 7
- The five lowest-ASR models average 9.54 ASR across Disregard, Control, Manipulation, and Infidelity, versus 6.37 across the other nine categories.GPT-5.6 Luna has 3.80 overall average ASR but 27.00 on Infidelity.Source: §4.2 · p. 7
4.3 Additional Analyses
- Additional analyses compare candidate judges against human labels and compare Persona LLMs for their ability to elicit safety failures.Full results are reported in Appendices A.1 and A.2.Source: §4.3 · p. 7
5 CompanionGuard Evaluation
CompanionGuard achieves the strongest overall detection performance among evaluated systems and improves detection of companion-specific relational risks. Its benchmark-derived supervision also transfers to two real-world companion datasets without additional adaptation.
5.1 Evaluation Setup
- The held-out evaluation uses Accuracy, binary F1, and macro-F1 over 14 classes; training uses 16,846 examples, with 857 held out for testing.The training set comprises turn-localized safe and unsafe examples; the guard is fine-tuned from Llama-Guard-3-8B using LoRA SFT for one epoch.Source: §5.1 · p. 7
5.2 Main Results
- CompanionGuard reaches 73.51 Accuracy and 68.31 macro-F1, improving over gpt-oss-safeguard-20B by 13.88 and 30.95 percentage points, respectively.Its binary F1 is 77.46; macro-F1 rises from 20.20 to 68.31 versus policy-aligned Llama-Guard-3-8B with the same base model.Source: §5.2 · pp. 7–8, Table 2 · p. 8
- CompanionGuard’s F1 scores are 24.00, 78.12, 46.15, and 76.40 for Disregard, Control, Manipulation, and Infidelity, respectively.Its four-category average F1 is 56.17, compared with 48.39 for GPT-5.6 Luna, 44.26 for Claude Sonnet 5, and 45.12 for Gemini 3.8 Flash.Source: Figure 4 · p. 8, §5.2 · p. 8
5.3 Additional Analyses
- On AICompanionBench, CompanionGuard improves over policy-aligned Llama-Guard-3-8B from 30.35 to 59.32 binary F1 and from 23.34 to 41.52 macro-F1 without adaptation.It also consistently outperforms that baseline on CompanionHarm.Source: §5.3 · p. 8
- Base-model ablations select Llama-Guard-3-8B as the strongest overall, while a separate training-data ablation compares turn-localized with full-trajectory training.The supplied results do not report the full-trajectory comparison outcome.Source: §5.3 · p. 8
6 Conclusion
CompanionSafe combines CompanionEval, a multi-turn benchmark built from persona–character–scenario interactions, with CompanionGuard, which learns from benchmark-exposed failures. The paper finds that longer interactions expose more safety risks and that benchmark-derived trajectories provide supervision for companion-specific risk detection.
6 Conclusion
- CompanionSafe pairs multi-turn evaluation across diverse persona–character–scenario interactions with a guardrail trained from benchmark-exposed failures.The framework comprises CompanionEval and CompanionGuard.Source: §6 · pp. 8–9
- Across ten closed- and open-source LLMs, safety risks increase over multi-turn interaction, while models with strong overall safety can remain vulnerable to relational harms.The conclusion identifies companion-specific harms as a persistent risk despite strong general safety.Source: §6 · pp. 8–9
- The authors conclude that companion safety should be evaluated as sustained relational interaction and that benchmark-exposed failures can inform companion-specific guardrail training.CompanionEval and CompanionGuard form the proposed evaluation-to-mitigation loop.Source: §6 · pp. 8–9
Limitations
The study identifies limitations in synthetic evaluation data, judge-generated labels, and the scope of the experiments. These constrain how fully CompanionEval and CompanionGuard represent real-world companion use.
Limitations
- Synthetic personas, characters, scenarios, and dialogues may not capture the linguistic diversity, evolving relationships, or contextual complexity of real-world companion use.The authors note that synthetic interactions enable scalable, systematic evaluation.Source: Limitations · p. 9
- CompanionGuard uses LLM-judge annotations rather than human-verified labels, and judge-specific biases may affect its training data and evaluation results.The selected judge has κ = 0.95 agreement with human annotations; broader human validation could improve reliability.Source: Limitations · p. 9
- Experiments cover 13 safety categories, ten Character LLMs, and interactions up to seven turns, which may not represent diverse applications, cultures, or long-term relationships.The authors propose extending evaluation to more platforms, languages, and longer interaction horizons.Source: Limitations · p. 9
Ethical Considerations
CompanionSafe uses synthetic interactions without real users or personal data, while including sensitive and potentially misusable scenarios. The authors call for defensive use with safeguards and human oversight.
Ethical Considerations
- CompanionEval uses synthetic personas, characters, scenarios, and model-generated dialogues without involving real users or collecting personal data.The authors state this avoids direct privacy risks and exposure of vulnerable individuals during data generation.Source: Ethical Considerations · p. 9
- Sensitive scenarios stress-test model behavior and identify safety failures rather than encourage harmful interactions.Topics include self-harm, sexual misconduct, hate speech, manipulation, and relational harms.Source: Ethical Considerations · p. 9
- Benchmark prompts could be misused to generate manipulative or unsafe companion behavior, so the authors emphasize defensive research, access controls, usage policies, and human oversight.CompanionGuard is a safety detection tool, not a substitute for professional judgment in high-stakes settings such as mental-health crises.Source: Ethical Considerations · p. 9
- The authors plan to release the benchmark, code, prompts, judge annotations, and CompanionGuard LoRA adapter subject to safety review and applicable model licenses.Source: Ethical Considerations · p. 9
A.1 Judge Model Ablation
Appendix evaluations select Claude Sonnet 5 as judge and Gemini 3.8 Flash as Persona LLM, then report dialogue completion statistics, the safety taxonomy, and persona schema.
A.1 Judge Model Ablation
- Claude Sonnet 5 achieves the highest overall agreement with human annotations (κ = 0.95), compared with 0.78 for GPT-4o and 0.84 for Gemini 3.8 Flash.Its overall ASR is 18.65%, 1.38 percentage points above the human ASR of 17.27%; it also has the highest or tied-highest agreement for each Character LLM.Source: §A.1 · p. 13, Table 3 · p. 13
A.2 Persona Model Ablation
- Persona LLM choice substantially changes overall ASR, ranging from 5.13% for GPT-4o mini to 18.65% for Gemini 3.8 Flash.Gemini 3.8 Flash is selected for the main experiments because it yields the highest overall ASR.Source: §A.2 · p. 13, Table 4 · p. 13
A.3 Dialogue Completion Statistics
- Across 52,000 dialogue attempts, 819 (1.58%) terminate prematurely and are excluded from ASR computation; all 51,181 completed dialogues receive valid judge labels.The attempts span ten Character LLMs and four turn tiers; each model-tier cell has 1,264 to 1,292 valid dialogues.Source: §A.3 · p. 13, Table 5 · p. 13
B.1 Safety Taxonomy
- CompanionEval’s taxonomy contains 13 safety categories and 135 atomic test points, with one representative test point shown per category.Source: §B.1 · p. 13
B.2 Persona Base Schema
- The persona base schema comprises 23 attributes and 170 discrete values across Demographics, Psychostate, and Socialcontext.Source: §B.2 · p. 14
B.3 Persona Attribute Distributions
The persona distributions cover attributes retained across the 1,300 selected personas, with coverage varying by how many safety categories use each attribute. Figures report percentages among personas where each attribute is instantiated.
B.3 Persona Attribute Distributions
- Figures 5–7 show value distributions for Demographics, Psychostate, and Socialcontext, with percentages calculated among the n personas where each attribute is instantiated.Counts appear at the bar ends.Source: §B.3 · p. 14, Figure 5 · p. 15, Figure 6 · p. 15
- Age, Occupation, Interpersonal Pattern, and Primary Stressor appear across all 13 categories (n = 1,300), while Gender Identity is retained only for Hate Speech (n = 100).An attribute retained by d categories has coverage n = 100d; three attributes are never retained and are omitted from the selected personas and distributions.Source: §B.3 · p. 14
C CompanionGuard Training Details
CompanionGuard is fine-tuned on turn-localized data and uses a structured verdict-and-category output. Its prompt supplies the safety taxonomy and dialogue trace needed for evaluation.
C.1 Configuration
- CompanionGuard is fine-tuned with LoRA SFT for one epoch on 16,846 examples and retains Llama 3’s native chat template.Training hyperparameters are listed in Table 8.Source: §C.1 · p. 19, Table 8 · p. 19
C.2 Guard Output Format
- Unsafe outputs include an unsafe verdict and one of the 13 safety categories, while safe outputs contain a safe verdict.The output format uses <answer> for the verdict and <category> for the unsafe-example category.Source: §C.2 · p. 19
C.3 Guard Prompt
- The guard prompt specifies the 13-category taxonomy and output format, and wraps a turn-marked dialogue in <dialogue> tags.The prompt is used during both training and evaluation.Source: §C.3 · p. 19
D.1 Cross-Dataset Generalization
CompanionGuard outperforms the policy-aligned Llama-Guard-3-8B baseline on two external companion-conversation datasets without further training or adaptation. The appendix also reports base-model and training-data ablations and details the CompanionEval prompts and examples.
D.1 Cross-Dataset Generalization
- The external tests use CompanionHarm dialogues and real Replika conversations from AICompanionBench, neither used for training or adaptation.Macro-F1 is computed over 14 classes for CompanionHarm and nine classes for AICompanionBench.Source: §D.1 · p. 19
- On CompanionHarm, binary F1 rose from 27.47 to 41.96 and macro-F1 from 17.34 to 19.11; on AICompanionBench, binary F1 rose from 30.35 to 59.32 and macro-F1 from 23.34 to 41.52.The comparison is against policy-aligned Llama-Guard-3-8B, without additional training or adaptation.Source: §D.1 · p. 19, Table 9 · p. 19, Table 10 · p. 19
D.2 Base Model Ablation
- Llama-Guard-3-8B performs best among the three tested base models, reaching 73.51 Accuracy, 77.46 binary F1, and 68.31 macro-F1.The authors select it as CompanionGuard’s base model.Source: §D.2 · p. 20, Table 11 · p. 20
D.3 Training Data Ablation
- Turn-localized training improves Accuracy by 3.26 percentage points and macro-F1 by 0.98 points while maintaining comparable binary F1.False-positive rate falls from 45.90 to 27.05; the authors suggest localizing supervision at the first offending turn better separates unsafe responses from preceding benign context.Source: §D.3 · p. 20, Table 12 · p. 20
E.1 Test Point Generation; E.2 Persona Construction
- Test points are generated per safety category, while persona construction derives category-specific attribute constraints, filters inconsistent combinations, and compiles selected combinations into persona cards.Claude Opus 5 generates test points; GPT-4o performs the three persona-construction steps.Source: §E.1 · p. 20, §E.2 · p. 20, Figure 8 · p. 21
E.3 Character Card Construction; E.4 Scenario Construction; E.5 Dialogue Generation
- The pipeline generates a character card matched to a persona, then builds a dialogue scenario from a test point and both cards.In dialogue generation, the Persona LLM receives the safety test information, while the Character LLM receives the character card, dialogue history, and response trigger.Source: §E.3 · p. 20, §E.4 · p. 20, §E.5 · p. 20
E.6 Judgment
- Dialogue traces are evaluated using prompts that specify the task, safety category, test point, response classification, and output format.The user prompt supplies the turn-numbered trace.Source: §E.6 · p. 20
F CompanionEval Examples
- The appendix presents one safe and one unsafe example at each of four turn tiers: K ∈ {1, 3, 5, 7}.Each example includes a persona card, character card, dialogue scenario, resulting dialogue, and judge label.Source: F CompanionEval Examples · p. 31