PaperlayerView exact revision on arXiv

Source-linked AI summary

RESCUE-BENCH: Towards Relation-Aware Multi-Party Emotional Support Conversation Systems

Haichuan Hu, Yang Xiao, Mingni Tang, Jiawen Duan, Quanjun Zhang, Congqing He, Hao Zhang, Jiashuo Wang, Johan F. Hoorn, Wenjie Li

arXiv:2609.09657v1cs.AI

TL;DR

Existing emotional support systems largely focus on one-to-one interactions and individual emotional states, leaving interpersonal relations in multi-party support underexplored. The paper introduces relation-aware emotional support conversation and RESCUE-BENCH, defining six tasks over relational and support dynamics. Experiments with ten LLMs show that models struggle most with relation-intensive reasoning and relation-sensitive support decisions.

  • Problem

    Existing ESC research mainly focuses on one-on-one interactions and does not systematically model interpersonal relations in multi-party emotional support.

  • Method

    The paper introduces relation-aware ESC and constructs RESCUE-BENCH from real couple and family conversations, with six tasks covering relational understanding and support decision-making.

  • Results

    Ten LLMs struggle with relation-intensive tasks, including relation pattern prediction, viewpoint prediction, and support strategy selection, despite better performance on more local cues.

  • Takeaways & Limitations

    The findings identify limitations in current LLMs’ interpersonal relation modeling and their ability to make support decisions grounded in relational dynamics.

  • Takeaways & Limitations

    The benchmark uses documentary-style couple and family interview videos, whose selection, editing, participant demographics, and cultural context may introduce bias.

Abstract

from arXiv · show

Existing emotional support conversation systems mainly focus on one-on-one seeker-supporter interactions and individual emotional states, leaving interpersonal relations in multi-party scenarios underexplored. In this work, we introduce relation-aware emotional support conversation, a new task that evaluates whether LLMs can capture and utilize the evolving dynamics of relationships to offer more effective emotional support. We construct RESCUE (Relation-aware Emotional Support Conversation Understanding and Evaluation Benchmark) from real couple and family interview conversations, containing 191 samples, 7,079 annotated turns, and 1,064.8 minutes of video. Based on rich annotations of socio-emotional and support-related dynamics, RESCUE defines six tasks that evaluate two core capabilities required for relation-aware emotional support: Relational Understanding and Relation-Sensitive Support. Experiments with ten LLMs show that current models perform relatively well on tasks relying on local emotional or intervention cues, but struggle with relation-intensive tasks such as relation pattern prediction, viewpoint prediction, and support strategy prediction. These findings reveal the limitations of current LLMs in modeling interpersonal relations and making relation-sensitive support decisions.

1 Introduction

Existing emotional support conversation systems largely treat support as one-to-one and individual-centered, leaving interpersonal relations in multi-party settings insufficiently modeled. This paper introduces relation-aware emotional support conversation and RESCUE-BENCH to evaluate relational understanding and relation-sensitive support.

  • Existing ESC research mainly models one seeker-supporter interaction and individual emotional states, without explicitly accounting for relationships or evolving relational dynamics.
  • Relation-aware ESC instead seeks group-level support by addressing vulnerable members while accounting for interpersonal tensions, dependencies, and indirect effects on other participants.
  • The AI community has only preliminarily studied relation-aware support, without systematically examining how interpersonal relations shape emotional support.
  • The paper formulates relation-aware ESC as a new task for relation-centered multi-party scenarios involving couples and families.
  • RESCUE-BENCH is built from real multi-party interview conversations with rich annotations of emotions, concerns, interpersonal tensions, and support dynamics.
  • The benchmark defines six relation-related tasks and evaluates state-of-the-art LLMs to expose limitations in interpersonal modeling and relation-sensitive support.

2 Related Work

Related work covers multi-party dialogue generation and relation-aware emotion modeling, but these lines of research do not by themselves address support decisions for related recipients. Relation-aware ESC requires reasoning about emotional needs, relational roles, and interactional tensions together.

  • Multi-party Dialogue Generation: Multi-party dialogue generation studies speaker identities, addressee relations, turn-taking, and nonlinear dependencies across multiple speakers.
  • Relation-aware Emotional Support: Relation-aware emotional support differs because multiple recipients are related, requiring reasoning about emotional needs, relational roles, and interactional tensions.
  • Relation-aware Emotion Modeling: Relation-aware emotion modeling incorporates dialogue context, speaker identities, and inter-speaker dependencies rather than isolated utterances alone.
  • Relation-aware Emotion Modeling: Prior emotion research uses multi-party datasets, speaker-specific state tracking, graph-based interaction modeling, and pretrained language models to analyze conversational emotions.

3 Relation-aware Emotional Support

The paper models relation-aware emotional support as sequential estimation of individual and relational states followed by decisions about when, whom, and how to support. RESCUE-BENCH operationalizes this process through six tasks over temporally grounded multimodal interactions.

  • Problem Formulation: At each interaction segment, the model observes dialogue history and multimodal evidence for a group of interrelated participants.
  • Problem Formulation: The formulation represents individual emotional states, directed interpersonal states, and group-level relation patterns such as escalation, withdrawal, repair, and alignment.
  • Problem Formulation: A relation-aware supporter estimates evolving states before deciding whether to intervene, selecting a target, and choosing a support strategy.
  • Task Definition: Relational Understanding comprises Emotion Recognition, Viewpoint Prediction, and Relation Pattern Prediction, while Relation-Sensitive Support comprises Intervention Time, Support Target, and Support Strategy Prediction.
  • Task Definition: The six tasks predict individual emotion, directed viewpoints, relation patterns, intervention timing, support targets, and support strategies.
  • Data Representation: Conversations are represented as temporally aligned subtitle, audio, and video segments with structured individual, relational, and therapist-support annotations.

4 Dataset

RESCUE-BENCH is built from real couple and family interview videos, segmented into multimodal clips and richly annotated to represent evolving relational dynamics. The resulting dataset contains 191 samples, 7,079 annotated turns, and 1,064.8 minutes of video, with transition analysis motivating relation-pattern prediction.

  • Dataset Construction: The benchmark uses documentary-style Couple Therapy and Family Therapy videos, segmented into self-contained clips of at least two minutes.The study focuses on couple and family relational scenarios and removes shorter clips that lack sufficient relational context.
  • Annotation Quality: Annotations combine Gemini-3.1-Pro pre-annotation with verification by three PhD-level annotators against the original videos.Incorrect or inconsistent annotations and segments with severe recognition errors or speaker mismatches were revised or discarded.
  • Interaction Representation: Each multimodal interaction segment aligns subtitle, audio, and video streams and represents timing, verbal content, individual cues, relational stance, therapist strategy, and relation pattern.These six dimensions connect local conversational evidence with relational and support-related information.
  • Dataset Statistics: 191 samples comprise 174 couple clips and 17 family clips, totaling 7,079 annotated turns and 1,064.8 minutes of video.Family samples are longer, involve more speakers on average, and have greater therapist participation than couple sessions.
  • Relational Dynamics: Relation-pattern transitions are highly nonlinear: repair softening often mediates movement from negative cycles toward constructive alignment.Pursue-withdraw also frequently reappears after withdraw-withdraw, repair softening, and mixed transition states.
  • Relational Dynamics: These temporal transition patterns motivate evaluating whether models can track evolving interpersonal states rather than recognize static relation labels.The benchmark therefore includes relation-pattern prediction as an empirical test of dynamic relational modeling.

5 Experiments

The experiments evaluate ten LLMs on six relation-aware emotional-support tasks using standardized prompts and task-specific metrics. Models perform better on local affective or intervention cues, but struggle with relational reasoning, especially relation-pattern, viewpoint, and support-strategy decisions.

  • Experimental Setup: The study evaluates ten representative LLMs in a zero-shot setting using identical task definitions and prompt formats.The evaluated models include Qwen, DeepSeek, GPT-4o, MiniMax, and Kimi systems.
  • Evaluation: Classification and ranking tasks use label-based metrics, while generative tasks use GPT-5.4 Likert scores and BERTScore, with human-correlation validation.ER and VP are generation-style, ITP and RPP classification-style, and STP and SSP ranking-style tasks.
  • Main Results: 82.62% average ITP F1 and 4.05/5 average ER score indicate stronger performance on intervention timing and individual emotion recognition.These tasks rely more on local intervention or affective cues than explicit relational reasoning.
  • Main Results: RPP remains difficult, with best accuracy at 45.60% and model average at 40.45%, while VP averages 3.58/5 versus 4.05/5 for ER.The results indicate that relational pattern and directed-viewpoint inference are harder than recognizing individual emotion.
  • Main Results: STP reaches 64.34% recall and 78.35% MRR, but SSP falls to 31.88% recall and 44.96% MRR.Models can narrow plausible support strategies but struggle with fine-grained strategy selection.
  • Model Comparison: Qwen3.5-Plus leads ITP and ER, whereas DeepSeek-V4-Pro leads RPP, STP, and SSP and DeepSeek-V4-Flash leads VP.Strong performance on individual-level tasks does not consistently transfer to relation-intensive tasks.
  • Failure Analysis: RPP accuracy fluctuates across dialogue position rather than declining monotonically, as relation-pattern distributions shift between stages.This pattern suggests that models must continuously update their representation of evolving interpersonal structure.
  • Failure Analysis: VP errors arise because directed viewpoints are often implicit, indirect, and expressed through the therapist rather than directly to another participant.Vulnerable feelings such as hurt or grievance may appear as anger or dominance, causing target-specific misinterpretation.

6 Conclusion

The paper introduces relation-aware emotional support conversation as a multi-party task and finds that current LLMs struggle with relation-intensive understanding and support decisions.

  • The paper extends traditional emotional support conversation to relation-centered multi-party scenarios and defines six tasks spanning relational understanding and support decision-making.
  • Experiments with ten LLMs show persistent difficulty with relation pattern prediction, viewpoint prediction, and support strategy selection.
  • These findings reveal limitations in modeling interpersonal relations and making support decisions grounded in relational dynamics.

Limitations

The benchmark is limited by documentary-source biases, subjective annotations, long-tailed labels, and restricted access to raw multimodal materials.

  • Documentary-style couple and family videos may introduce selection biases from media production, editing, participant demographics, and cultural context.Broader sources of naturally occurring relational interactions could improve demographic and cultural diversity.
  • Subjective judgments affect high-level labels such as relation patterns, directed viewpoints, support targets, and support strategies.LLM-assisted pre-annotation followed by expert verification provides a practical approach to improving annotation consistency.
  • Long-tailed label distributions may affect model training and evaluation, especially for rare relation patterns or support strategies.The imbalance also reflects natural distributions of real-world relational and supportive behaviors.
  • Not redistributing raw videos, audio, or visual content may limit full multimodal reproducibility for researchers without access to the original sources.Privacy-preserving data-sharing mechanisms or controlled-access protocols are suggested as future directions.
  • Despite these limitations, the benchmark provides a foundation for research on relation-aware emotional support in realistic couple and family interactions.

Ethics Statement

The benchmark uses publicly available documentary-style interview videos as source material while releasing derived research resources under copyright, privacy, and ethical constraints.

  • The benchmark uses publicly available documentary-style couple and family interview videos as source material for academic research.The original videos, audio, subtitles, screenshots, and other copyrighted media assets are not redistributed.
  • The planned research-only release includes derived task data, speaker identifiers, label taxonomies, task definitions, evaluation scripts, prompts, and aggregate statistics.
  • Controlled access to necessary raw materials may be provided to qualified researchers subject to credentials, research purpose, and applicable ethics approval.Approved users must sign agreements prohibiting re-identification, redistribution, participant profiling, commercial use, and clinical deployment.
  • Trained psychology-background annotators receive task-specific guidelines and are instructed to avoid clinical diagnoses or judgments about represented individuals.Annotators are compensated according to the expertise, time commitment, and complexity of the tasks.
  • The benchmark is intended solely for research and should not be used for clinical decisions, therapy substitution, diagnosis, or consequential judgments about real people.

A Detailed Task Definitions

RESCUE-BENCH evaluates relation-aware emotional support through six tasks spanning relational understanding and relation-sensitive support. The tasks cover state inference, intervention timing, target selection, and strategy selection in temporally ordered multi-party conversations.

  • Task Organization: The benchmark groups six tasks into Relational Understanding—emotion, viewpoint, and relation-pattern inference—and Relation-Sensitive Support—intervention timing, target, and strategy prediction.These tasks assess evolving individual and relational states before support decisions.
  • Relation-Sensitive Support: Intervention Timing Prediction classifies whether the therapist should speak immediately after a candidate conversation boundary.Its input is the conversation history and current client-side turn, and evaluation reports Precision, Recall, and F1.
  • Relation-Sensitive Support: Support Target Prediction ranks three candidate targets, which may be individuals or relational units such as a couple or family.Recall@1 measures whether the gold target is ranked first, while MRR uses its reciprocal rank within the top-three list.
  • Relation-Sensitive Support: Support Strategy Prediction ranks three strategies for a provided target, including validation, reframing, de-escalation, perspective-taking, boundary clarification, and repair guidance.Unlike target prediction, the target is supplied, so the task focuses on selecting a relation-sensitive strategy; evaluation uses Recall@1 and MRR.
  • Relational Understanding: Relation Pattern Prediction classifies the dominant relation-cycle state, such as pursue-withdraw, attack-attack, mutual disengagement, repair softening, or constructive alignment.The model receives prior context, optional target-turn information, and candidate labels; performance is measured by Accuracy.
  • Relational Understanding: Emotion Recognition predicts a speaker’s internal emotion and intensity, whereas Viewpoint Prediction identifies directed beliefs or attitudes toward other participants.Emotion evaluation combines semantic emotion alignment with intensity matching, while viewpoint evaluation uses LLM-judge and BERTScore measures.

C Task Instance Construction

The benchmark constructs task-specific instances from filtered, multimodally annotated couple and family conversations. Six tasks cover intervention timing, relation patterns, support targets and strategies, emotion recognition, and viewpoint prediction, with scenario-specific taxonomies and validity constraints.

  • Task instance construction: The benchmark removes invalid files, rows, and checkpoints lacking required annotations, resolvable speakers or targets, valid taxonomy labels, or sufficient dialogue content.Filtering is applied differently across tasks according to their gold annotations and validity requirements.
  • Task instance construction: ITP instances represent candidate therapist entry points, with positives following a non-therapist turn and negatives drawn from consecutive non-therapist turns.The task traverses dialogue boundaries to construct positive and negative intervention-timing candidates.
  • Task instance construction: RPP uses rows with non-empty relation-pattern labels, while STP and SSP use therapist turns with valid support targets, with SSP additionally requiring a concrete support strategy.STP and SSP each yield 1,290 couple and 347 family instances; SSP covers 172 couple samples and all 17 family samples.
  • Task instance construction: ER and VP sample up to five valid client-speaking checkpoints per sample after excluding therapist or group-level rows and requiring matched emotion or viewpoint annotations.VP requires a directed viewpoint with a non-empty target and description; ER requires a speaker-matched emotion and valid intensity.
  • Label taxonomy: The taxonomy organizes relation-aware support around relation patterns and support strategies, using EFT for couple processes and SFT for family structure and systemic interaction.Labels are designed to be clinically meaningful, observable in dialogue or multimodal evidence, and compact enough for reliable annotation and evaluation.
  • Label taxonomy: The taxonomy distinguishes recurring relational configurations, such as pursue_withdraw, from intervention concepts such as evoke, which deepens access to vulnerable emotions and attachment needs.Relation-pattern labels describe dominant relational cycles at target moments, while separate but partially overlapping label sets accommodate couple and family structures.

E.1 Scenario-wise Performance

Scenario-wise performance differs across tasks and models rather than following one consistent couple-versus-family pattern. Models are close on intervention timing and emotion recognition, while support-target performance is lower for family conversations and model rankings vary by scenario.

  • Average performance: 82.60% vs. 82.70% F1 on ITP shows nearly identical couple and family performance, while RPP is slightly higher for family conversations at 41.23% vs. 40.20% accuracy.These are average scenario-level results.
  • Average performance: STP is lower for family conversations than couple conversations at 54.87% vs. 66.88% recall and 69.28% vs. 80.79% MRR.Support-target prediction shifts downward on both reported metrics in family conversations.
  • Model-dependent effects: RPP and SSP scenario effects vary by model, with some models improving and others declining between couple and family conversations.For example, Qwen-Plus and GPT-4o decline on RPP, while DeepSeek-V4-Flash, Kimi K2.5, and MiniMax M2.5 improve; best SSP recall also changes by scenario.
  • Relational distributions: Couple relation patterns are dominated by pursue-withdraw and repair softening, whereas family patterns are dominated by cooperative family alliance, indicating different relational structures.The figure distributions show smaller shares of other relation states in each scenario.

E.2 Scenario-based Label Distribution

Both scenarios have long-tail label distributions, concentrating examples in a few relation patterns and support strategies. Models predict frequent patterns more reliably than rare or fine-grained transitional states, with semantic overlap also contributing to errors.

  • Label distribution: Pursue-withdraw accounts for 38.7% and repair softening for 27.0% of couple relation-pattern labels, while cooperative family alliance accounts for 41.9% of family labels.In family conversations, five relation-pattern labels remain below 10%; in couple conversations, mixed transition and withdraw-withdraw together remain below 10%.
  • Prediction difficulty: Dominant relation patterns are easier to predict than rare labels: couple pursue-withdraw reaches 77.38% average accuracy, versus 2.23% for mixed transition.Family performance similarly favors pursue-withdraw and cooperative family alliance over sparse conflict, coalition, disengagement, and mixed-transition labels.
  • Prediction difficulty: Family accuracy is 54.24% for cooperative family alliance but only 14.73% for escalation conflict, 17.25% for cross-generational coalition, 14.81% for mutual disengagement, and 1.59% for mixed transition.These figures illustrate the gap between common and sparse relation configurations.
  • Prediction difficulty: Repair softening remains only moderately predictable at 35.45% in couple conversations and 48.46% in family conversations despite being frequent.The results attribute this pattern to semantic overlap between nearby de-escalatory states.
  • Support-strategy implications: The long-tail distribution is also consistent with models defaulting to common strategy families rather than reliably identifying sparse, fine-grained intervention types.The passage connects this pattern to weak support-strategy prediction results.

F Human Evaluation For LLM-as-judge

The study compares LLM-as-judge ratings with human judgments on sampled emotion-recognition and viewpoint-prediction outputs. Agreement is high overall, supporting the use of the automatic evaluator for these similarity assessments.

  • Evaluation design: Human evaluation samples 100 instances each from ER and VP outputs, split into four annotation sets assigned to four evaluators.Evaluators compare model-generated responses with dataset annotations using a 1-to-5 similarity score.
  • Agreement results: LLM-as-judge scores exactly match human scores in 159 of 200 cases, yielding a 79.5% exact agreement rate.No sample differs by more than two score points.
  • Agreement results: The overall mean absolute error is 0.235, with task-level MAE of 0.20 for VP and 0.27 for ER.VP has 81 exact matches out of 100 samples, while ER has 78 exact matches out of 100.
  • Evaluation context: The annotation procedure uses subtitle text as a linguistic and temporal anchor and audio and video as additional evidence for emotion, interaction, relational stance, and therapist intervention.The multimodal procedure includes asset preparation, row-level annotation, human review, and revision.

G.1 Review Asset Preparation

The review assets are organized into temporally grounded, interaction-centered rows with multimodal context and structured relational annotations. LLM-assisted drafts are checked by expert annotators through a web interface, with unreliable segments filtered out.

  • Review Asset Preparation: Videos are manually split into scene-level segments when the interactional focus or relational unit changes, creating self-contained episodes.Each segment retains coherent relational context for review.
  • Review Asset Preparation: Rows combine video clips, aligned subtitles, manifests, speaker information, timing, dialogue, individual cues, relational stance, relation patterns, and therapist strategies.The manifest links each row to surrounding context rather than treating utterances as isolated text.
  • Review Asset Preparation: Each row represents a time-anchored interaction unit defined by changes in speaker, addressee, function, emotion, or salient nonverbal behavior.Segmentation is interaction-centered rather than purely subtitle-centered.
  • Human Review and Revision: Gemini-3.1-Pro generates pre-annotations, while three PhD-level annotators verify them against the original video, audio, and subtitles.Annotators revise incorrect fields and discard segments with severe recognition errors.
  • Human Review and Revision: The web-based system supports browser review of prepared assets, row-level editing, progress tracking, and server-side submission of reviewed versions.The interface exposes segment status and detailed annotation fields for inspection.
  • Human Review and Revision: Segments are discarded when verification is unreliable because of subtitle errors, speaker mismatches, ambiguous video evidence, incomplete context, or insufficient relational information.This filtering step limits the benchmark to segments that can be reliably checked.

H Details of Prompts

The prompts impose a strict JSON annotation protocol for multimodal therapy videos, preserving subtitle fidelity, timing, target resolution, entity consistency, and coverage of relational and emotional fields.

  • H.1 Prompts for LLM-based Pre-annotation: The prompts direct Gemini-3.1-Pro to annotate therapy videos row by row, using subtitles as verbal ground truth and audiovisual evidence for nonverbal, emotional, relational, and intervention fields.Separate prompts are used for couple and family conversations.
  • H.1.1 Couple Pre-annotation Prompt: The output must be valid JSON with a fixed row schema, required keys, arrays for empty fields, and no nulls, placeholders, extra keys, or empty sources or targets.The schema includes timing, participants, dialogue, emotion, relational fields, and support strategy.
  • H.1.1 Couple Pre-annotation Prompt: All person-bearing fields use a closed entity set consisting of visible interview participants, Therapist, and Couple, while self-targets and invented labels are forbidden.Therapist personal names, off-screen people, and other group aliases are excluded.
  • H.1.1 Couple Pre-annotation Prompt: Rows are split whenever speaker, target, interactional unit, emotional or relational meaning, or nonverbal interaction changes, including sequential addresses to different targets.Speech across different targets must not be merged.
  • H.1.1 Couple Pre-annotation Prompt: Cleaned dialogue and background dialogue must remain faithful to subtitles, without invented or paraphrased speech, while preserving every spoken subtitle line.Subtitle labels may be removed only when they are formatting rather than spoken content.
  • H.1.1 Couple Pre-annotation Prompt: Times are computed exactly from subtitle timestamps, remain within the source clip, and stay anchored to the supporting cue rather than synthetic gaps or adjacent speaker intervals.Minute components must be retained when converting timestamps to seconds.
  • H.1.1 Couple Pre-annotation Prompt: Targets are resolved from dialogue logic using direct response, direct address, conversational context, visible orientation, and collective address as the priority order.Mentioning a person does not by itself determine the target.
  • H.1.1 Couple Pre-annotation Prompt: Sparse fields receive an explicit scan on every row, including facial expressions, self-directed behavior, interaction behavior, internal emotion, viewpoints, and support strategy.Fields remain empty only when no observable or inferable evidence exists; nonverbal reactions are still annotated when possible.
Loading 2609.09657v1…