PaperlayerView exact revision on arXiv

Source-linked AI summary

Are they lovers or friends? Evaluating LLMs' Social Reasoning in English and Korean Dialogues

Eunsu Kim, Junyeong Park, Juhyun Oh, Kiwoong Park, Seyoung Song, A. Seza Doğruöz, Alice Oh, Najoung Kim

arXiv:2510.19028v3cs.CL

TL;DR

Social relationship inference from dialogue is uncertain and context-dependent, yet earlier evaluations often used simplified setups. The paper introduces SCRIPTS, a bilingual movie-dialogue benchmark with likelihood-based labels, and evaluates nine models. Results show limited social reasoning, frequent UNLIKELY predictions, and minimal benefit from CoT prompting or thinking models.

  • Problem

    Earlier social-relationship evaluations often use simplified settings, although dialogue-based relationship inference is uncertain and context-dependent.

    Source: §1 · p. 1

  • Method

    SCRIPTS is a benchmark of 1.1k English and Korean movie-script dialogues with three likelihood-based relationship labels, used to evaluate nine LLMs.

    Source: §1 · p. 1

  • Results

    Nine models show limited social reasoning and often predict UNLIKELY relationships; even GPT-4o produces UNLIKELY labels in 10.9% of cases and LESS LIKELY labels in 10.0%.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    CoT and thinking models do not consistently benefit social reasoning, and the analysis identifies where models fail, especially on UNLIKELY predictions.

    Source: §1 · p. 2, §7 · p. 9

    The analysis covers only English and Korean, and movie-script data may not fully reflect real-world conversation or generalize to other cultural contexts.

    Source: Limitations · p. 9

Abstract

from arXiv · show

As LLMs are increasingly deployed in real-world interactions, their social reasoning in interpersonal communication becomes critical. To explore their capabilities, we introduce SCRIPTS, a 1.1k-dialogue dataset in English and Korean, sourced from movie scripts and propose a social reasoning task based on SCRIPTS that evaluates the capacity of LLMs to infer the social relationships (e.g., friends, lovers) between speakers in each dialogue. Evaluating nine models on our task, current LLMs achieve around 75--80% on the English dataset and 58--69% in Korean, and models predict an Unlikely relationship in 10--25% of responses in both languages. Furthermore, we find that thinking models and chain-of-thought prompting provide minimal benefits for social reasoning and occasionally amplify social biases. In sum, there are significant limitations in current LLMs' social reasoning capabilities, especially for Korean, highlighting the need for efforts to develop socially-aware LLMs across languages.

1 Introduction

The paper introduces SCRIPTS to evaluate social relationship reasoning in English and Korean dialogues, addressing the ambiguity and limited realism of earlier task setups. Across nine models, it finds limited performance and frequent socially implausible predictions, while chain-of-thought prompting and thinking models offer little benefit.

1 Introduction

  • Earlier studies often use simplified classification setups, limited relationship types, or simple dyadic conversations, despite relationship inference being uncertain and context-dependent.A remark such as “You never listen to me” may be a serious complaint between romantic partners or playful banter between friends.Source: §1 · p. 1
  • SCRIPTS contains 1.1k English and Korean movie-script dialogues, with HIGHLY LIKELY, LESS LIKELY, and UNLIKELY labels that represent different degrees of plausibility.The soft labels distinguish nonsensical responses from plausible but less prominent ones.Source: §1 · pp. 1–2
  • CoT prompting and thinking models provide minimal social-reasoning benefits and occasionally amplify social biases.Lower-performing Llama-3.1-8B-Instruct makes HIGHLY LIKELY predictions in only 41.3% of the English dataset.Source: §1 · p. 2
  • Qualitative analysis identifies four failure modes across English and Korean, while adding partner background, formality, hierarchy, and intimacy information reduces UNLIKELY outputs.The reported effect includes mitigating nonsensical responses.Source: §1 · p. 2
  • Nine evaluated models show limited social reasoning, often predict UNLIKELY relationships, and vary substantially in their HIGHLY LIKELY predictions.The introduction also reports frequent socially implausible inferences across both languages.Source: §1 · pp. 1–2

2 Related Work

Prior work on computational social relationship reasoning often simplifies the task, while linguistic and cultural context can shape the evidence available in dialogue. This paper contrasts those approaches with multi-turn, open-ended evaluation and motivates cross-linguistic assessment.

2 Related Work

  • Earlier studies often use classification or multiple-choice tasks, single utterances, movie summaries, or global character-pair labels that may not be evident in one dialogue.These choices can limit contextual variation and fail to capture how relationships are expressed in conversation.Source: §2 · p. 2
  • SCRIPTS uses multi-turn dialogues with multiple human-inferred labels and open-ended generation rather than fixed-choice classification.This allows a wider range of possible relationships and better reflects social complexity.Source: §2 · p. 2
  • Korean conveys relational information through terms of address, kinship terms, titles, pronouns, and a grammaticalized honorific system, unlike English.The honorific system conveys roles, status, and formality through verbal morphology.Source: §2 · pp. 2–3

3 SCRIPTS: Evaluating LLMs’ Interpersonal Social Reasoning

SCRIPTS evaluates interpersonal relationship inference in multi-turn English and Korean dialogues, using movie-script interactions and context-aware labels that represent uncertainty.

3.1 The Importance of Social Relationship Reasoning in LLMs

  • Contextual relationship errors can produce inappropriate responses, such as mistaking a store clerk-customer exchange for a mother-child relationship and encouraging oversharing.The paper frames social relationship reasoning as recognizing relevant relationships among conversation participants.Source: §3.1 · p. 3, §3 · p. 3

3.2 Dataset Design

  • SCRIPTS labels relationships within dialogue context and uses HIGHLY LIKELY, LESS LIKELY, and UNLIKELY tiers to represent plausibility.The dataset draws on movie scripts to capture varied natural interactions; Table 1 reports 1.1K dialogues from 60 movies and 230+ unique relationship types.Source: §3.2 · p. 3, Table 1 · p. 3
  • 19% of movie-level labels were irrelevant to a given dialogue, while applicable dialogues averaged more than three HIGHLY LIKELY relationships.These findings motivate dialogue-level labels that capture relationships in context rather than relying on static movie metadata.Source: §3.2 · p. 3

3.3.1 Collecting Human Annotations

Phase 1: Labeling UNLIKELY Relationships

  • UNLIKELY labels required at least two of three annotators to flag a relationship as contradicted; HIGHLY LIKELY labels were open-ended, and LESS LIKELY covered remaining types.Annotators also recorded demographic attributes and relational dimensions such as formality, hierarchy, and intimacy for downstream social-cue analyses.Source: Phase 1: Labeling UNLIKELY Relationships · p. 4

3.3 Dataset Construction

  • SCRIPTS contains 1,147 dialogues—580 English and 567 Korean—retained from 1,322 scenes after removing 13.2% for low annotation reliability.The dialogues came from 60 movie scripts, and the collection includes three-speaker scenes while inferring a relationship for a randomly selected speaker pair.Source: Table 1 · p. 3, §3.3 · p. 4, Phase 1: Labeling UNLIKELY Relationships · p. 5

4 Evaluating LLMs with SCRIPTS

The evaluation tests nine LLMs on open-ended relationship inference in SCRIPTS. GPT-4o performs best on both language datasets, while chain-of-thought and internal thinking offer no consistent advantage, and Korean-specialized models show mixed rankings.

4 Evaluating LLMs with SCRIPTS

  • The evaluation covers nine LLMs and uses open-ended generation, five runs per dialogue, majority responses, and HIGHLY LIKELY and UNLIKELY rates.GPT-4o evaluates short-form answers, achieving 92.0% human-validated accuracy in a validation experiment.Source: §4 · p. 5

4.1 Overall Performance

  • GPT-4o achieves the best HIGHLY LIKELY rate: 79% in English and 69% in Korean.Models incorrectly infer an UNLIKELY relationship in 10.8–31.9% of responses, with Korean rates higher by an additional 7.2–16.5%p.Source: §4.1 · p. 5

4.2 Does Thinking Help?

  • CoT does not consistently help, and internal thinking has mixed effects that are negligible or harmful in Korean.Thinking-model comparisons used one run per dialogue, unlike Figure 2’s five; none of the differences was statistically significant (bootstrap test, p > 0.05).Source: §4.2 · p. 6, Table 3 · p. 6

4.3 Do Korean-specialized models perform better on Korean dialogues?

  • A.X-4.0-Light and Qwen-3-8B rank first and second in both languages, but the remaining model rankings differ between English and Korean.Llama-3.1-Instruct-8B ranks third in English and last in Korean; Exaone-4.0-32B and Kanana-1.5-8B rank third and fourth in Korean.Source: §4.3 · p. 6, Table 4 · p. 6

5 Reasons Behind Failure of LLMs

The analyses identify failures in interpreting address and reference terms, integrating multiple cues, recognizing atypical relationships, and handling Korean-specific linguistic and cultural features. In 30 GPT-4o failure cases, the dominant error type differs by language.

5 Reasons Behind Failure of LLMs

  • Models can mistake a term of reference for a term of address, as when “Dad” triggers a parent–child inference despite the speaker calling the other person “Officer.”Korean models may also misread self-referential uses such as “the teacher” as referring to a third party.Source: §5 · pp. 6–7
  • Models struggle to integrate multiple cues, especially when their combination is atypical.The paper describes social relationship reasoning as identifying and combining contextual cues to reach a conclusion.Source: §5 · p. 7
  • Models sometimes prioritize one cue over contextual evidence: GPT-4o reads compliance with an instruction as hierarchy despite casual Korean speech without honorifics.In the example, the ground truth is friend/coworker, while Qwen, Gemini, and GPT predict supervisor–subordinate.Source: §5 · p. 7

Failure to Recognize Atypical Relationships

  • Models may reject human-annotated parent–child relationships when the exchange seems peer-like, reflecting stereotyped expectations about family roles.All human annotators agree on the relationship in the example, but the models favor a sibling interpretation or otherwise dismiss parent–child.Source: Failure to Recognize Atypical… · p. 7
  • Korean-specific confusions include interpreting kinship expressions such as eomeoni, Hyungsoo, and Hyungnim as evidence of the wrong relationship, and misreading honorifics.The paper reports that language- and culture-specific features account for the largest share of Korean failure cases.Source: Failure to Recognize Atypical… · pp. 7–8
  • 36.7% of English failures involved cue aggregation, while 46% of Korean failures involved language- or culture-specific features.The analysis examined 30 GPT-4o failure cases.Source: Figure 3 · p. 7, Failure to Recognize Atypical… · p. 8

6 Does Providing Additional Social Information Help?

The experiments test whether demographic cues and relational dimensions help models infer social relationships, using human-provided or model-generated information. Human-provided information reduces UNLIKELY predictions without consistent performance gains, while model-generated information is often inaccurate.

Experimental Setting

  • The study tests age and gender cues, relational dimensions, or both, supplied either as human-annotated labels or model-generated predictions.Four models are evaluated, excluding thinking-enabled and Korean-specialized models.Source: §6 · p. 8

Results With Ground Truth Labels

  • Human-provided relational information yields no substantial or consistent performance gains for GPT-4o, but reduces UNLIKELY predictions.Figure 4 reports results across six English-dataset settings; the paper says the tendency holds across models except Qwen-3-8B.Source: §6 · p. 8, Figure 4 · p. 8

Results With Model-Generated Labels

  • GPT-4o predicts age and gender with under 60% accuracy and relational dimensions with below 75% accuracy, limiting the reliability of model-generated auxiliary labels.The paper reports that such labels do not consistently help because they are often inaccurate.Source: §6 · pp. 8–9
  • Instructing models to infer social factors before relationships is ineffective, although demographic cues and relational dimensions can facilitate reasoning when accurately inferred.The authors note that models may over-rely on provided labels rather than context, and that label-driven changes do not always yield correct reasoning.Source: §6 · pp. 8–9

7 Conclusion

The paper introduces SCRIPTS to study social relationship reasoning across English and Korean. Its analyses find suboptimal performance and frequent UNLIKELY inferences, while suggesting explicit cues, broader exposure to atypical relationships, and language- and culture-specific approaches as directions for improvement.

7 Conclusion

  • SCRIPTS is a bilingual dataset introduced to investigate current LLM limitations in social relationship reasoning.The paper’s conclusion describes its experiments as showing that most models perform suboptimally in English and Korean and often infer UNLIKELY relationships.Source: §7 · p. 9
  • Chain-of-thought does not consistently benefit social reasoning, and the paper reports that reasoning techniques can occasionally amplify social biases.The conclusion also highlights frequent UNLIKELY relationship inferences.Source: §7 · p. 9
  • The authors suggest explicit relational cues at inference and broader exposure to rare, atypical, and culturally diverse relationships during training.They also point to language- and culture-specific approaches, citing improved Korean-dataset performance by Korean-specific models.Source: §7 · p. 9

Limitations

The study is limited to English and Korean movie-script dialogue, and its analysis of chain-of-thought traces may not reveal the mechanisms behind final answers.

Limitations

  • The analysis covers only English and Korean and may not generalize to other cultural contexts.The authors identify more languages as a target for future benchmark extensions.Source: Limitations · p. 9
  • Movie-script dialogues may not fully reflect real-world conversation, though the authors describe them as a practical proxy given privacy and labeling challenges.They suggest future work use more realistic sources, including privacy-preserving real conversations and human-AI dialogue logs.Source: Limitations · p. 9
  • Chain-of-thought traces may be post-hoc rationalizations rather than the mechanisms that produced the final answers.This caveat applies to the study’s analysis of those traces.Source: Limitations · p. 9

Ethics Statement

The study uses human annotation of anonymized scenes from selected English and Korean movie scripts to build SCRIPTS, with dialogue-level relationship labels and auxiliary social dimensions. The authors report IRB approval, informed consent, and participant compensation.

Ethics Statement

  • The study used preexisting movie scripts that may contain harmful or offensive content, and received KAIST IRB approval with informed consent from all participants.Annotators were paid hourly rates of KRW 15,000 in Korea and USD 20 in the U.S., approximately 1.5 times local minimum wage.Source: Ethics Statement · p. 9

A Dataset

  • SCRIPTS collection proceeds through movie selection, scene collection, OCR and human verification, filtering, anonymization, and relationship annotation.The appendix describes these as the dataset’s six collection steps.Source: A Dataset · p. 12

A.1 Movie Selection

  • Movie selection favored modern-day films after 2000 and excluded English films above PG-13 and Korean films above the 15-and-older rating.The collection covered 60 movies: 28 English and 32 Korean.Source: §A.1 · p. 12

A.2 Raw Scene Collection and Processing

  • Korean script dialogue was extracted with OCR, cleaned by GPT-4o, and checked by human annotators; the process yielded 16k English and 7k Korean scenes.Korean scripts were collected from physical copies at the Korean Film Archive and OCR-processed.Source: §A.2 · p. 12

A.3 Scene Filtering

  • Filtering selected 1,322 scenes from an initial 23k, requiring at least four utterances and two to three speakers while prioritizing unique character pairs.The selected set comprised 698 English and 624 Korean scenes.Source: §A.3 · p. 12

A.4 Anonymization

  • Character names were replaced with placeholders and any names missed by automation were manually anonymized to mitigate contamination and bias.Source: §A.4 · p. 13

A.5 Human Annotation

  • The project recruited 17 English and 14 Korean annotators, using sample-item screening, orientation, and training before annotation.Korean annotators were native speakers; English annotators were U.S. citizens or had lived in the United States for over ten years.Source: §A.5 · p. 13
  • Annotators selected likely relationship categories, marked unlikely relationships, and could add up to five open-ended likely labels; auxiliary dimensions were rated on five-point scales and collapsed to three-point labels.For multi-label relationship sets, agreement was measured with mean pairwise Jaccard similarity.Source: §A.5 · pp. 13–14

A.6 Diversity of SCRIPTS

SCRIPTS spans shared and language-specific relationship types and captures variation in intimacy, formality, and hierarchy. The section also describes the model setup, prompting, and evaluator validation used in the analyses.

A.6 Diversity of SCRIPTS

  • Social, organizational, and familial relationships are frequent among the ten most common relationship types in both language datasets.Figure 6 presents the ten most common relationships for each language.Source: §A.6 · p. 14
  • The datasets also contain language-specific roles and lexicalizations, including fine-grained Korean kinship relations and English family roles such as co-parents and babysitter–child.The authors connect these patterns to cultural context shaping relationship granularity and salience.Source: §A.6 · p. 14
  • Over 40% of friend relationships deviate from the typical intimate, informal, equal pattern across relational dimensions.Figure 7 compares intimacy, formality, and hierarchy distributions for six relationship types shared across the languages.Source: §A.6 · p. 14, Figure 7 · p. 15

B.1 Model Configuration

  • The evaluation uses nine models, including GPT-4o, o3, Gemini-2.5-Flash, and models from Qwen, Llama, A.X, Kanana, and Exaone.The listed model configuration sets temperature to 1.Source: §B.1 · p. 15

LLM-as-a-Judge Prompt

  • The LLM judge checks each model-predicted relation against supplied possible and impossible relationship sets.It marks whether each relation belongs to either set.Source: §B.2.1 · p. 15

B.2.2 Inference Prompt

  • The inference prompt asks models to infer the speakers’ relationship from dialogue, using listed examples when applicable and otherwise describing the relationship themselves.The chain-of-thought condition appends “Think step by step.”Source: §B.2.2 · pp. 15–16

B.2.3 Prompts Used in §6

  • The §6 prompts add age and gender or relational-dimension information, using either human gold labels or model-inferred labels.The model-generated setting first asks the model to infer the added information, then inserts it into the relationship prompt.Source: §B.2.3 · p. 16

B.3 Validating LLM-as-a-Judge

  • GPT-4o’s evaluator judgments were verified as correct on 97.85% of sampled English responses and 86.2% of sampled Korean responses.Two authors independently checked 100 question–answer pairs per language; their judgments matched on 96% of data points, with Cohen’s κ = 0.58.Source: §B.3 · p. 16

B.4 Results

The supplied results passages identify the scope and metric definitions of the performance tables, but do not provide their cell values. They also include an original Korean dialogue excerpt.

B.4 Results

  • Table 7 compares model performance in English and Korean, defining HIGHLY LIKELY as majority-response accuracy and UNLIKELY as the error rate for unlikely outputs.Source: Table 7 · p. 17
  • Table 8 reports English and Korean performance comparisons with chain-of-thought prompting and gives deltas in parentheses.Source: Table 8 · p. 17
  • Table 9 reports Korean-specialized model performance in both English and Korean.Source: Table 9 · p. 17

C.1 Original Korean Dialogue

  • The appendix includes an original Korean dialogue excerpt as an example.Source: §C.1 · p. 17

C.2 What cues do LLMs rely on in social relationship reasoning?

The qualitative CoT analysis finds that LLMs use address and reference, conversational setting, emotional tone, and relational dimensions to infer relationships; stereotyped expectations about hierarchy can also lead to failures.

C.2 What cues do LLMs rely on in social relationship reasoning?

  • LLMs use address terms, self-identification, and shared third-party titles to infer family, professional, or colleague relationships and task-oriented context.Examples include “Daddy,” professional titles, and speakers who share a formal title for a third party.Source: §C.2 · pp. 17–18
  • Conversation settings such as workplaces or homes help models infer relationships and intimacy between dialogue participants.Models also use emotional tone: friendly or supportive expressions are associated with intimacy, while angry or neutral expressions inform judgments of intimacy or formality.Source: §C.2 · p. 18
  • Models consider intimacy, hierarchy, and formality, but stereotyped expectations about parent-child hierarchy or marital equality can fail when interactions depart from those norms.A playful exchange alongside personal disclosure can lead models to infer a close relationship such as friendship or siblinghood.Source: §C.2 · p. 18

D Does Providing Additional Relational Dimension Help?

Associations Between relational dimension and Social Relationship Reasoning Performance

  • Most relational factors are positively associated with social relationship reasoning performance, and models’ age, gender, and relational-dimension inference performance also tends to align with overall reasoning performance.The paper presents these findings as associations from separate logistic regressions, not as causal effects.Source: Associations Between relational… · p. 18, Table 13 · p. 20
Loading 2510.19028v3…