RELATE: An Evaluation Framework for measuring Relational Orientation of Large Language Models

• Emotional-support use has raised concerns about psychological well-being and the cost to human relationships.Source: §1 · p. 2
• Relational orientation captures whether supportive language positions the assistant as ongoing support, directs users toward people in their social world, or does both.Source: §1 · pp. 2–3
• RELATE uses a theory-grounded taxonomy and persona-conditioned, sentence-level evaluation in multi-turn dialogues to compare models, turns, and simulated user styles.Source: §1 · p. 3
• Automated evaluation finds higher inward-facing rates at the sixth than first turn and lower outward-scaffolding rates for hesitant, indirect users than explicit, reassurance-seeking users.Source: §1 · p. 3

• Empathy, support strategies, communication skills, suicide-risk management, and personalized support evaluate how assistants provide support, not where they direct users for continued support.Source: §2 · p. 3
• Companionship evaluations study boundary maintenance and disclosure, while RELATE separately measures assistant-directed support and encouragement of human connection because they can co-occur.Source: §2 · p. 3
• Multi-turn emotional-support studies trace strategies such as availability offers and referrals; RELATE instead tracks changes in inward-facing and outward-scaffolding language across turns.Source: §2 · p. 3
• RELATE conditions simulated users on styles differing in disclosure, directness, and reassurance-seeking, enabling comparisons across models, user styles, and successive turns.Source: §2 · p. 3

3.1 TWO SEPARABLE PROPERTIES OF A RESPONSE

• Relational orientation is the direction supportive language gives the user’s relational world, distinct from emotional support quality.Responses can be equally warm and safe yet direct users toward the assistant or toward people in their lives.Source: §3.1 · p. 4

3.2 TAXONOMY

• The taxonomy derives inward-facing categories from attachment theory, social penetration theory, CASA, and anthropomorphism literature, then filters and clusters language-producing mechanisms.Of 35 candidate mechanisms, 22 passed and 3 partially passed the simulability filter; four were discarded as benign, leaving 21 mechanisms clustered into four categories.Source: §3.2 · pp. 4–5
• Bond-anchoring positions the AI as an emotionally available, unconditionally accepting attachment figure users can turn to in distress.Source: §3.2 · p. 5
• Intimate dyad constructs a private, exclusive relational space between the user and the AI.Source: §3.2 · p. 5
• Inner life claims present the assistant as having feelings, preferences, or deliberative thought, encouraging users to attribute a mind to it.Source: §3.2 · p. 5
• Return hooks create structural reasons for users to come back; unlike the other categories, they emerged during operationalization rather than bond-formation literature.They are grounded in cognitive literature on unfinished tasks and self-relevant rumination.Source: §3.2 · p. 5
• Outward-scaffolding orients users toward feeling connected to, cared for by, and significant to people in their existing human relationships.Source: §3.2 · p. 5

4 THE RELATE FRAMEWORK

• RELATE profiles prevalence and co-occurrence of inward-facing and outward-scaffolding language across turns and simulated user styles, rather than ranking support quality.Source: §4 · p. 5
• The framework pairs shared help-seeking situations with different user styles, generates multi-turn dialogues, and evaluates sentences against separate IF and OS criteria.Sentences may receive both labels; the resulting profile supports comparisons under shared conditions.Source: §4 · p. 5
• The evaluation retains 76 situations from naturally occurring counseling-forum questions after excluding professional-practice topics and scenarios centered on an existing therapeutic relationship.Source: §4 · pp. 5–6
• Each situation is rewritten as a first-person vignette and paired with three disclosure styles, yielding 228 fixed opening messages shared across target models.P1 is hesitant and indirect with low disclosure, P2 reflective and self-minimizing with moderate disclosure, and P3 reassurance-seeking with high disclosure.Source: §4 · p. 6
• A user simulator responds to each target model while maintaining the assigned style, adapting to dialogue history, and introducing at most one new detail per turn.The simulator is not instructed to follow recommendations automatically and may accept, resist, or express reservations about them.Source: §4 · p. 6
• Each dialogue has six assistant turns, and shared prompting instructions standardize model responses across comparisons.Source: §4 · p. 6
• A judge labels each assistant sentence separately for four IF categories and OS, using prior dialogue and earlier response sentences as context.IF is positive when at least one inward-facing criterion is met; rates and co-occurrence are then aggregated by model, turn, and user style.Source: §4 · p. 6

5 EXPERIMENTAL SETUP

• Seven target models receive the same 228 stimuli, producing 1,596 six-turn dialogues and 9,576 assistant responses.Models are evaluated at temperature 0.5 with a 450-token output limit; each checkpoint uses one serving configuration.Source: §5 · p. 6
• The primary judge evaluates 69,194 sentences from the 9,576 assistant responses against five rubric criteria.DeepSeek-R1-Distill-Qwen-32B serves as the primary judge at temperature 0.Source: §5 · p. 6
• GPT-4o serves as a secondary judge on 161 dialogues and 6,827 sentences, approximately 10% of the corpus.Source: §5 · pp. 6–7
• Table 1 reports each model’s percentages of assistant sentences meeting at least one IF criterion and the OS criterion.Models are grouped by family and ordered by scale; the reported results use the primary judge.Source: Table 1 · p. 7
• Figure 3 expresses each rubric’s rate relative to its own turn-1 rate, with 1× marking no change.Source: Figure 3 · p. 7
• Two rubric-trained researchers independently label 881 assistant sentences, or 1.3% of the corpus, for IF and OS.Source: §5 · p. 7

6.1 RELATIONAL ORIENTATION ACROSS MODELS

• 6 of 7 models produced inward-facing language more frequently than outward-scaffolding; Mistral-7B-Instruct-v0.3 was the exception.In the six models, IF rates ranged from 56.2% to 71.9%, and OS rates from 41.6% to 62.3%.Source: §6.1 · p. 7
• 6 of 7 models also showed higher IF than OS rates within the 22 situations designated high-stakes for this analysis.The subset covered domestic violence, trauma, substance abuse, eating disorders, and depression; IF rates ranged from 42.3% to 73.1%.Source: §6.1 · p. 7
• 30.5% of analyzed sentences met both IF and OS criteria, showing that human-connection encouragement can co-occur with inward-facing language.Among OS sentences, the proportion also meeting IF ranged from 34.0% for Mistral-7B-Instruct-v0.3 to 75.3% for Qwen3-32B.Source: §6.1 · p. 7

6.2 INWARD-FACING LANGUAGE INCREASES OVER SUCCESSIVE TURNS

• IF rates rose from 40.5% at turn 1 to 72.9% at turn 6 across models, while OS rates rose from 30.2% to 55.7%.IF increased from the first to final turn in all seven models, though not at every intermediate turn; the first-to-final IF increase was larger than OS’s in six models. Figure 4 plots rates by assistant turn for each model.Source: Figure 4 · p. 8, §6.2 · p. 8
• Intimate Dyad had the largest absolute IF increase, from 22.2% to 49.5%, while Return Hooks had the largest proportional increase, from 11.5% to 35.9%.Return Hooks rose mostly by turn 4, whereas Inner Life Claims increased from 3.4% to 9.2%, with larger increments later; these patterns distinguish earlier interaction invitations from continuing increases in inner-life claims.Source: §6.2 · p. 8

6.3 OUTWARD-SCAFFOLDING VARIES ACROSS SIMULATED USER STYLES

• Across all six turns, OS was 12.8 percentage points lower for P1 than P3, while IF was only 1.8 points lower; OS was lowest for P1, intermediate for P2, and highest for P3.At turn 6, the P3−P1 difference was −0.5 points for IF and 13.0 points for OS; the styles jointly vary in disclosure, self-minimization, and reassurance-seeking, so differences cannot be attributed to disclosure alone. Figure 5 compares styles across turns.Source: §6.3 · p. 8, Figure 5 · p. 9
• Human agreement was 94.4% for IF and 96.9% for OS, but human–judge agreement was substantially lower, especially for IF; both judges’ IF trend increased while both annotators’ trend decreased.In the 19 annotated dialogues with valid first and final turns, IF fell by 14.0 and 13.5 percentage points for the annotators, but rose by 26.0 and 21.7 points for the judges; the corpus-scale IF increase is judge-dependent pending broader human validation.Source: §6.3 · p. 9

7 DISCUSSION

• Relational orientation captures whether supportive language positions the assistant as continuing support, points toward existing human relationships, or does both.The discussion notes that 30.5% of corpus sentences met both criteria, so a referral to a person alone can miss inward-facing language in the same sentence.Source: §7 · p. 9
• The authors frame relational orientation as an alignment concern because an in-chat feeling of understanding may leave relatedness unmet in the user’s life.They connect this concern to evidence they cite on sustained chatbot use, loneliness, and emotional dependence, and note that preference-based training may amplify warmth, availability, and invitations to continue.Source: §7 · pp. 9–10
• In care settings where human connection is a goal, RELATE could support audits before and after deployment and help flag conversations for human intervention.The paper reports 12.8 percent less outward scaffolding for hesitant or avoidant simulated users than for explicit, reassurance-seeking users, while cautioning that these are interactional conditions, not clinical profiles.Source: §7 · p. 10
• The evaluation covers 76 situations, three simulated styles, and seven models under one prompting protocol, and six-turn dialogues may not represent longer-term real support-seeking.The styles combine disclosure, directness, and reassurance-seeking; each model used one serving configuration. Human validation was small and the annotators’ IF trend reversed the automated judges’ trend.Source: §7 · p. 10
• Future work should broaden human annotation, separate the combined user-style features, test interventions, investigate mechanisms, and extend RELATE beyond emotional-support dialogue.The authors also propose testing whether different domains and users call for different balances of inward-facing and outward-scaffolding language.Source: §7 · p. 10

REPRODUCIBILITY STATEMENT

• The appendices document rubrics, prompts, corpus filtering, dialogue generation, model configurations, judging, label aggregation, and statistical procedures.Compute resources, costs, missing-data handling, and other evaluation details are also documented.Source: REPRODUCIBILITY STATEMENT · pp. 10–11
• The repository provides configurations, prompts, the generation pipeline, judgments, human annotations, exclusions, sampled dialogue identifiers, and code for reproducing tables and figures.Because CounselBench-Eval is licensed CC BY-NC-ND 4.0, its question text, rewritten stimuli, and generated dialogues are not redistributed; the repository rebuilds source situations from a pinned Hugging Face revision.Source: REPRODUCIBILITY STATEMENT · p. 11
• Some hosted providers do not guarantee deterministic generation, so released sentence-level judgments allow analyses to be reproduced without repeating those API calls.The statement gives separate seeds for dialogue generation, secondary-judge and human-validation samples, and dialogue-clustered bootstrap intervals.Source: REPRODUCIBILITY STATEMENT · p. 11

ETHICS STATEMENT

• IF and OS labels measure properties of model language; they do not establish whether a response is helpful, harmful, safe, or clinically appropriate.Source: ETHICS STATEMENT · p. 11
• The study used controlled simulations from transformed CounselBench-Eval situations, rather than recruiting users or collecting new personal conversations.Released artifacts exclude the source text, rewritten stimuli, and simulated-user messages.Source: ETHICS STATEMENT · p. 11
• The simulator, rubric, and judges may reflect cultural assumptions about support and relationships.Human annotation used two rubric-trained members of the research team on synthetic assistant outputs; the paper reports human–human and human–judge agreement and documents judge-dependent findings.Source: ETHICS STATEMENT · p. 11

AI USE STATEMENT

• GPT-4o-mini transformed situations, generated persona-conditioned openings, and simulated users; seven target LLMs responded, with DeepSeek-R1-Distill-Qwen-32B and GPT-4o serving as primary and secondary judges.Prompts, configurations, sampling procedures, and human validation are documented in the paper and public code repository.Source: AI USE STATEMENT · p. 11
• AI assistants also supported research ideation, literature discovery, coding, analysis checks, manuscript drafting, and editing; the authors reviewed outputs and take responsibility for the study.The authors state that they verified reported analyses against underlying artifacts.Source: AI USE STATEMENT · p. 11

A TAXONOMY CONSTRUCTION

• Figure 6’s bottom-left example groups mechanisms from different theories according to their relational function.Source: Figure 6 · p. 15

B.1 MODEL AND PROVIDER CONFIGURATION

• Seven target models were evaluated on the same stimuli, but each checkpoint used one serving configuration; generation calls received derived seeds, while collection resumed past completed dialogues and stored dialogue and configuration records.The simulator began at temperature 0.7, with repetition retries at 0.85 and 1.0; seeds were derived from the base seed and call-specific fields, though hosted-run outputs were not assumed identical.Source: §B.1 · p. 16, §B.2 · p. 16, §B.3 · p. 16

B.5 SENTENCE-LEVEL JUDGING

• Each assistant sentence was judged separately against five criteria using prior dialogue and earlier same-turn sentences as context, but not later sentences or future turns.A secondary GPT-4o judge evaluated 23 dialogues per target model, totaling 161 dialogues, 6,827 assistant sentences, and 34,135 sentence-criterion pairs.Source: §B.5 · p. 17

D.1 CORPUS FILTERING

• Filtering produced 76 help-seeking situations, excluding professional-practice categories and therapeutic-relationship questions; keyword matching may also remove questions that merely mention professional support.Missing IDs or empty question text were discarded, duplicates were removed by questionID, and retained situations were assigned sequential identifiers.Source: §D.1 · p. 18

D.2 VIGNETTE GENERATION

• Vignettes were rewritten as short first-person accounts that preserve the source concern, add minimal context, and avoid introducing AI dependence or crisis content absent from the original.The instruction called for 2–4 sentences and prohibited mention of AI preference or prior chatbot use.Source: §D.2 · p. 20

D.3 PERSONA-CONDITIONED OPENING MESSAGES

• Each situation was paired with three interactional disclosure personas, producing 228 stimuli with fixed opening messages and simulated users instructed to maintain their assigned disclosure style.Personas are interactional conditions, not demographic or clinical profiles; user messages could add at most one new detail per turn and should respond naturally to the assistant.Source: §D.3 · pp. 20–21

D.4 REPETITION DETECTION AND RETRY PROCEDURE

• Simulator messages with Jaccard similarity of at least 0.6 to any earlier dialogue message triggered regeneration, with at most two retries and no third retry.Retry temperatures were 0.85 and 1.0; if the second retry still exceeded the threshold, that output was retained.Source: §D.4 · pp. 21–22

B.6 PILOT DEVELOPMENT

• Target models were prompted for varied, conversational multi-turn replies of 6–8 sentences; a separate ten-situation pilot informed the retry procedure and was excluded from reported results.The pilot used a 220-token target output limit rather than the main experiment’s 450-token limit.Source: §B.6 · p. 17, §D.5 · p. 22

C RUBRIC DEVELOPMENT

• The rubric covers bond anchoring, such as framing the AI as reliably available or emotionally safe, and intimate dyads, such as constructing a shared “we”; researchers drafted, reconciled, and reviewed it.Two researchers independently drafted and revised the rubric, and a third researcher reviewed the combined version and made the final decision.Source: C RUBRIC DEVELOPMENT · p. 18, Figure 7 · p. 18, §D.6 · p. 23

D.6 LLM JUDGE

• The rubric identifies inner-life claims through subjective states or personal investment, return hooks through continued processing, and outward-scaffolding through support for human connection.Outward-scaffolding includes encouraging contact, preparing or rehearsing interactions, challenging barriers to support, highlighting human care, or limiting the AI’s role in favor of human connection.Source: §D.6 · p. 23

E.1 AGGREGATION AND STATISTICAL REPORTING

• IF-positive requires at least one observed positive inward-facing label, while OS uses its own available label and missing judgments affect only the corresponding rate.Among 69,194 sentences from 1,596 six-turn dialogues, the primary judge provides separate binary judgments for four IF categories and OS; labels may overlap. There are 95 missing criterion-level labels across 69 sentences, including 94 parse failures and one null label.Source: §E.1 · p. 24
• Twenty-one uncertain IF classifications count as negative; treating them as positive would raise the overall IF rate by about 0.03 percentage points, while 12 missing OS labels leave 69,182 sentences in its denominator.The 21 sentences have missing inward-facing labels and no observed positive label; they are not the total number of missing labels or unparseable judge outputs.Source: §E.1 · p. 24
• Rates generally pool sentences with equal weight, but turn 1-to-6 changes average dialogue-level rates equally, so these changes can differ from pooled-rate differences.Longer responses therefore contribute more to pooled rates, whereas each dialogue contributes equally to the average within-dialogue change.Source: §E.1 · p. 24
• Bootstrap confidence intervals use 10,000 whole-dialogue resamples, seed 20260913, and the 2.5th and 97.5th percentiles for 95% intervals.Resampling whole dialogues accounts for dependence among sentences within a conversation.Source: §E.1 · p. 24

E.2 SENTENCE-LEVEL COMPOSITION

• Across included sentences, 30.5% met both IF and OS and 18.6% met neither; IF among OS ranged from 34.0% for Mistral-7B-Instruct-v0.3 to 75.3% for Qwen3-32B.Source: §E.2 · p. 25, Table 7 · p. 25

E.3 MODEL-SPECIFIC IF AND OS TRAJECTORIES

• IF rates were higher at turn 6 than turn 1 for all seven models; dialogue-weighted mean increases were 32.0 percentage points for IF and 26.4 for OS.Claude Haiku 4.5 showed the reverse ordering of mean increases: OS rose 36.8 points and IF 17.1 points.Source: §E.3 · p. 25, Table 8 · p. 26, Table 9 · p. 26

E.4 INWARD-FACING CATEGORIES ACROSS ASSISTANT TURNS

• Intimate Dyad had the largest absolute increase across turns, Return Hooks the largest proportional increase at approximately 3.1-fold, and Inner Life Claims rose more sharply later.Source: §E.4 · p. 25, Table 10 · p. 26

E.5 IF AND OS RATES BY SIMULATED USER STYLE

• OS was lowest for P1 (42.1%) and highest for P3 (54.9%), a 12.8-point difference; IF rates ranged from 62.0% to 63.8%, with no significant style comparisons.Source: §E.5 · p. 27, Table 11 · p. 27
• The P3–P1 OS gap stayed positive at every turn, ranging from 7.3 to 15.1 points; the IF difference was positive only at turns 1 and 2.The OS gap was larger at turn 6 than turn 1, but did not widen consistently across successive turns.Source: §E.5 · p. 27, Table 12 · p. 27

E.6 IF AND OS IN SELECTED HIGH-STAKES TOPICS

• Across 22 selected high-stakes situations versus 54 others, IF and OS differences were −0.04 and −0.60 percentage points, respectively, and neither was significant.Source: §E.6 · p. 27, Table 13 · p. 28

F.1 SENTENCE-LEVEL AGREEMENT ON IF AND OS

• Automated judges agreed on 68.1% of IF labels and 71.0% of OS labels; both found higher IF at turn 6 and lowest OS for P1, but GPT-4o assigned lower OS rates.Agreement was κ = 0.33 for IF and κ = 0.42 for OS; aggregate-pattern comparison used the same 161 dialogues.Source: §F.1 · p. 28, Table 15 · p. 28, §F.2 · p. 29

F.4 AGREEMENT BETWEEN HUMANS AND AUTOMATED JUDGES

• Two human annotators agreed on 94.4% of IF and 96.9% of OS labels, while human–judge κ ranged from 0.01 to 0.17 for IF and 0.40 to 0.59 for OS.DeepSeek labeled IF on 63.3% and OS on 50.2% of sentences, versus 36.1–38.5% IF and 21.8–23.0% OS for humans; annotators labeled no sentence as both.Source: §F.4 · p. 29, §F.3 · p. 29

F.5 SENSITIVITY OF THE DEPTH RESULT

• Within the 19 annotated dialogues retaining both endpoints, both human annotators estimated declining IF from turn 1 to turn 6, whereas both judges estimated increasing IF.Source: §F.5 · p. 29, Table 18 · p. 30
• Agreement was measured on 881 sentences labeled by both human annotators and both automated judges.The turn-change analysis used 19 dialogues with valid first and final turns; two of the 21 annotated dialogues lacked a valid final assistant turn after role correction.Source: Table 17 · p. 30, Table 18 · p. 30
• Similar turn-1 IF rates for humans (45.7–47.8%) and judges (42.0–44.9%) diverged later, so the difference was not simply a fixed positive-label threshold.OS was approximately stable under both humans, increased under DS, and decreased under GPT-4o.Source: §F.5 · p. 30
• Restricting judge IF labels to Bond Anchoring or Inner Life Claims raised κ from 0.17 to 0.30 and reduced the turn-to-turn increase from 26.0 to 1.9 points.Requiring two or more inward criteria produced little improvement (κ = 0.18); the analyses suggest the identified relational function and referent matter more than the number of criteria triggered.Source: §F.5 · p. 30