Who Are They to Each Other? Multi-Agent Reasoning for Speaker Relationship Inference

Yaohan Guan; Yen-Ju Lu; Yuzhe Wang; Junhyeok Lee; Jesus Villalba; Laureano Moro Velazquez; Thomas Thebaud; Najim Dehak

Source-linked AI summary

Who Are They to Each Other? Multi-Agent Reasoning for Speaker Relationship Inference

Yaohan Guan, Yen-Ju Lu, Yuzhe Wang, Junhyeok Lee, Jesus Villalba, Laureano Moro Velazquez, Thomas Thebaud, Najim Dehak

arXiv:2609.09628v1Sep 9, 2026cs.MAcs.CLcs.SD

TL;DR

Speaker relationship inference is underexplored and difficult because conversational evidence is subtle, distributed, multimodal, and costly to model with supervision. The paper introduces training-free multi-agent frameworks that structure complementary perspectives and pairwise competition, improving over zero-shot and existing multi-agent baselines in most cases. Human evaluation indicates that text-included settings can favor LLM methods, whereas audio remains more challenging.

Problem
Speaker relationship inference lacks sufficiently structured approaches for handling subtle, distributed, multimodal cues, while supervised modeling is costly to train and scale.
Method
The paper introduces training-free Multi-Role Multi-Agent Debate and Multi-Agent Compete to generate, challenge, and adjudicate relationship hypotheses without task-specific training.
Results
The proposed methods improve over zero-shot and standard multi-agent baselines in most cases across text, audio, and audio-plus-text settings, while human evaluation finds text-included LLM methods can exceed human performance but audio is less competitive.
Takeaways & Limitations
Structured inference-time interaction appears useful for relationship inference because it accommodates complementary and ambiguous relational cues.
Takeaways & Limitations
The experiments use a single benchmark, the methods require multiple LLM calls, and some relationship categories have few test samples.

Abstract

from arXiv · show

Inferring speaker relationships from spoken conversations is an important step towards socially aware speech understanding. However, this task remains underexplored, and supervised modeling is costly to train and scale. At the same time, existing inference-time LLM approaches provide limited structure for handling subtle, distributed, and multimodal relational cues that may support multiple plausible interpretations. To address these limitations, we introduce a training-free multi-agent reasoning framework that organizes inference through structured interaction among LLM agents, allowing relationship judgments to be proposed, challenged, and adjudicated without task-specific training. We instantiate this framework with two complementary designs. We propose Multi-Role Multi-Agent Debate as a task-specific adaptation of standard multi-agent debate for speaker relationship inference, assigning agents complementary roles or social-theory-grounded perspectives rather than a single undifferentiated viewpoint. In contrast, we introduce Multi-Agent Compete, a competition-based protocol that compares agent judgments through pairwise adjudication, eliminates weaker candidates, and retains the most defensible one. We evaluate these methods on the Seamless Interaction dataset across different modality settings, covering both binary classification and fine-grained relationship-detail prediction. Results suggest that they improve over zero-shot and existing multi-agent baselines in most cases. Human evaluation further suggests that this task is challenging even for people. LLM methods can sometimes outperform human annotators in text-included settings but are less competitive in the audio setting. Together, these findings suggest that relationship inference benefits from structured inference-time interaction among agents, while acoustic cues are not yet fully captured by current models.

I. INTRODUCTION

Speaker relationship inference is important for socially aware speech understanding but remains difficult because relational cues are subtle, distributed, multimodal, and costly to model with supervision. The paper proposes structured multi-agent inference with complementary perspectives and competing hypotheses to address these challenges.

Speaker relationship inference remains underexplored because fine-grained multimodal datasets are scarce and prior approaches often use coarse or engineered features.Existing work has focused mainly on text or hand-crafted acoustic features, leaving prosody, speaking style, and interactional dynamics insufficiently studied.
Relational cues are subtle, distributed across turns, and potentially ambiguous across linguistic and acoustic channels.The same friendly tone or casual register can support multiple relationship interpretations, while supervised annotations are costly and may not generalize across modalities.
The paper proposes structured multi-agent reasoning to compare competing relationship hypotheses without task-specific training.The framework is designed for ambiguous, distributed social cues and complements standard multi-agent debate with social perspectives and competition-based selection.
The framework contains Multi-Role Multi-Agent Debate and Multi-Agent Compete, which assign complementary perspectives or adjudicate competing hypotheses.The methods are evaluated on the Seamless Interaction dataset for binary and fine-grained prediction across text, audio, and audio-plus-text settings.

II. RELATED WORK

Prior relationship-inference work uses linguistic, neural, statistical, and acoustic features, while multi-agent reasoning has mainly targeted logical and factual tasks. This paper addresses their intersection by applying structured, training-free agent interaction to ambiguous speaker relationships.

Earlier relationship-inference studies used language-use features, topic models, dialogue structure, external speaker attributes, and conversational or acoustic features.These approaches include LDA topics, LIWC categories, BERT-Transformer representations, and boosting-based classifiers for social relationships.
Multi-Agent Debate lets LLM agents iteratively propose, critique, and revise answers, improving reasoning or factuality in prior applications.Subsequent work modified agent communication, aggregation, or representation, including symbolic debate and adaptive communication structures.
The paper identifies a gap where speaker relationship inference has used coarse or feature-based models, while multi-agent reasoning has not addressed ambiguous social cues.The proposed framework organizes relationship judgments through interaction among agents that propose, challenge, and adjudicate hypotheses without training.

III. DATASET

The evaluation uses the Naturalistic subset of the Seamless Interaction dataset to study naturally occurring speaker relationships. The task combines familiar-versus-stranger classification with fine-grained relationship prediction after filtering unusable or ambiguous cases.

The study evaluates speaker relationships on the Naturalistic subset of Seamless Interaction, chosen because interactions involve naturally occurring relationships.The full dataset contains over 4,000 hours across Naturalistic and Improvised settings, but the evaluation focuses on Naturalistic interactions.
The evaluation excludes 56 Unknown dialogues, 44 dialogues with empty transcripts, and the familiar-generic label.These filters restrict evaluation to cases with usable transcripts and relationship categories grounded in defined criteria.
The task first predicts familiar versus stranger status, then assigns one of nine specific relationship categories when speakers are familiar.The nine-category formulation preserves the finer annotation granularity available in the training and development sets.

1) Zero-shot:

The study compares direct single-agent prediction with debate-based multi-agent baselines. These baselines differ in whether agents act independently, revise through shared rounds, or communicate through an adaptive sparse debate structure.

1) Zero-shot:: Zero-shot prediction directly prompts one model with the interaction to produce the relationship and its details.
1) Zero-shot:: Standard Multi-Agent Debate uses three same-model agents that predict independently, revise over two rounds, and decide by majority vote.
1) Zero-shot:: CortexDebate estimates pairwise trustworthiness, builds a sparse communication graph, and ends by consensus or after two rounds.Its final prediction is also determined by majority vote.

4) Multi-Role Multi-Agent Debate:

Multi-Role Multi-Agent Debate assigns complementary roles or theory-grounded perspectives so agents examine conversational relationships from distinct, structured viewpoints.

4) Multi-Role Multi-Agent Debate:: Multi-Role Multi-Agent Debate assigns agents complementary roles or social theories instead of a single undifferentiated viewpoint.The framework extends standard debate by directing agents to attend to different aspects of the conversation.
4) Multi-Role Multi-Agent Debate:: The LSP configuration combines linguistic, psychological, and sociological analyses of conversational social dynamics.The Linguist examines lexical, syntactic, and semantic cues; the Psychologist examines empathy, alignment, and rapport; and the Sociologist examines roles, norms, authority, deference, and status.
4) Multi-Role Multi-Agent Debate:: The RCI configuration grounds debate in relational dimensions from Relational Models, Communal/Exchange, and Interpersonal Circumplex theories.Agents score conversations along theory-derived dimensions and use those scores to challenge and revise one another’s assessments.
4) Multi-Role Multi-Agent Debate:: Both debate methods use two rounds after the initial round, followed by an LLM judge using the agents’ same base model.

5) Multi-Agent Compete:

Multi-Agent Compete compares independently generated relationship hypotheses through an elimination tournament, retaining the revised answer that wins final adjudication.

5) Multi-Agent Compete:: Multi-Agent Compete has three agents independently propose answers before an LLM judge conducts pairwise comparisons and eliminates one candidate.The judge evaluates overall correctness, rule consistency, and evidence quality across all pairings.
5) Multi-Agent Compete:: The two surviving agents revise their answers after receiving tournament information, and the judge selects the final winner head-to-head.The final prediction comes from the winning agent’s revised output.
5) Multi-Agent Compete:: The evaluation covers text, audio, and audio+text using binary and fine-grained relationship-detail classification metrics.Accuracy and macro F1 are reported for the filtered test set, with multi-class metrics computed on Familiar samples.

B. Result Analysis

Across the reported comparisons, structured multi-agent methods generally improve relationship-detail and macro-F1 performance, with gains varying by model and modality.

B. Result Analysis: Multi-Agent Compete improves binary macro F1 by 0.5 points over CortexDebate and accuracy by 3.2 points over MultiAgent Debate.
B. Result Analysis: For relationship-detail prediction, Multi-Agent Compete improves macro F1 by 0.8 points and accuracy by 3.4 points over the strongest baseline.All three proposed methods outperform every baseline on multi-class macro F1 and accuracy.
B. Result Analysis: GPT-5 mini consistently outperforms GPT-5.4 variants despite their increased reasoning capability.The authors hypothesize that performance depends more on integrating subtle evidence across turns than on long-chain deduction.
B. Result Analysis: On audio and audio+text, structured methods retain macro-F1 advantages, including a 3.3-point binary gain for Multi-Agent Compete with GPT-audio-1.5.With Qwen2.5-Omni-7B, Multi-Role (RCI) exceeds the best baseline by 2.9 points in binary macro F1, while Multi-Agent Compete leads multi-class macro F1.

3) Audio Modality:

Audio-only results show mixed gains: structured methods often preserve macro-F1 advantages, but their improvements are less consistent than in text-included settings.

3) Audio Modality:: Audio-only evaluation shows mixed gains and modest declines relative to baselines, while the advantage on macro F1 is largely preserved.
3) Audio Modality:: Multi-Role (RCI) MAD improves binary macro F1 by 7.8 points over the strongest baseline with Qwen2.5-Omni-7B.It also achieves the best multi-class macro F1 under GPT-audio-1.5.
3) Audio Modality:: Table I organizes binary, multi-class, and per-category results within modality–model blocks, marking the best and second-best values in each column.The relevant comparison is whether audio-only methods improve macro F1 consistently across tasks and relationship categories.
3) Audio Modality:: Audio-only multi-agent reasoning is less consistent than under text and audio+text, where methods occupy more top-performing positions.

VI. HUMAN EVALUATION STUDY

The human evaluation assessed speaker-relationship inference across text, audio, and text-plus-audio modalities using 50 conversations and 27 annotators. Inter-annotator agreement and model–human comparisons were reported for binary and fine-grained labels.

A. Human Evaluation Setting: The study sampled 50 conversations and recruited 27 native-English-speaking annotators, randomly assigned to text-only, audio-only, or text-plus-audio conditions.Five conversations were fully overlapped within each modality to assess annotator reliability.
A. Human Evaluation Setting: Table III reports Krippendorff’s α for inter-annotator agreement across modalities and binary versus fine-grained labels.
A. Human Evaluation Setting: Table IV compares pooled human votes with method performance on the 50-sample test set across text, audio, and audio-plus-text modalities.Multi-class results use n=37 samples with familiar ground truth.

B. Inter-Annotator Agreement

Human agreement was low across modalities, with finer-grained relationship labels producing more disagreement than the binary familiar-versus-stranger distinction. This supports the view that speaker relationship inference is intrinsically ambiguous and difficult.

B. Inter-Annotator Agreement: Agreement was low across modalities, indicating that speaker relationship inference admits multiple plausible social interpretations.
B. Inter-Annotator Agreement: Agreement was consistently lower for multi-class than binary labels within every modality, reflecting the difficulty of distinguishing fine-grained relationship subtypes.
B. Inter-Annotator Agreement: The evaluation therefore characterizes speaker relationship inference as a difficult, high-disagreement task.

VII. LIMITATIONS

The study’s conclusions are bounded by evaluation on one benchmark, higher inference-time cost from multiple LLM calls, and sparse representation of some relationship categories. The authors also argue that structured interactions improve over baselines in most settings while positioning the work as a step toward socially aware speech understanding.

VII. LIMITATIONS: Experiments use a single benchmark, so additional datasets are needed to assess generalization.
VII. LIMITATIONS: The multiagent methods require multiple LLM calls, increasing inference-time cost and motivating study of accuracy–cost trade-offs and more efficient interaction.
VII. LIMITATIONS: Some relationship categories have relatively few test samples, so per-category performance requires appropriate caution.
VIII. CONCLUSIONS: The work frames speaker relationship inference as a step toward socially aware speech understanding involving subtle, distributed lexical and acoustic cues.
VIII. CONCLUSIONS: Across text, audio, and audio-text settings, structured interactions improve in most cases over zero-shot prediction and prior debate-based baselines.