PaperlayerView exact revision on arXiv

Source-linked AI summary

Conversation Is a Two-Body Problem: Dyadic Evaluation of Full-Duplex Dialogue Models

Sungnyun Kim, Sungwoo Cho, Jihwan Oh, Se-Young Yun

arXiv:2610.08125v1eess.AScs.CL

TL;DR

Full-duplex dialogue benchmarks commonly use fixed interlocutors, although turn-taking and other behaviors arise through two speakers influencing one another. DyaFDB evaluates role-assigned models in direct dyadic conversations and finds that performance depends on the partner, task, and willingness to engage.

  • Problem

    Full-duplex benchmarks commonly evaluate one model against fixed or agenda-bound interlocutors, while turn-taking and interruption are joint products of coupled speakers.

    Source: §1 · p. 2

  • Method

    DyaFDB has two role-assigned full-duplex models converse directly across four tasks, with both sides scored offline and partner model and role varied.

    Source: §1 · pp. 2–3, §3.1 · p. 5

  • Results

    Across tasks, performance depends on partner behavior, task structure, and disposition to engage; the same defender leaks its secret in 39–63% of conversations depending on the attacker.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    A full-duplex agent should be tested in realistic interactions across diverse opponents rather than evaluated only with a single fixed interlocutor.

    Source: §1 · p. 2, §6 · p. 10

    Current scores do not distinguish holding a role by ignoring or steering a partner, or winning through enforcement versus persuasion.

    Source: §I · p. 23

Abstract

from arXiv · show

Full-duplex spoken dialogue models listen and speak at the same time, enabling voice agents to have natural, low-latency interactions that turn-based systems cannot offer. However, they are commonly evaluated against single-sided interlocutors: pre-recorded audio that cannot react, or an automated examiner that reacts in real time but only administers a fixed sequence of tests and is never graded. These single-sided frameworks evaluate only half of a two-body problem, where turn-taking, overlap, and interruption are joint products of two coupled speakers. We propose DyaFDB, a framework that evaluates full-duplex models in a dyadic setup: two models converse directly under assigned roles with cooperative or conflicting goals, and both sides are scored offline with an external judge. DyaFDB probes how the two models behave toward each other, such as how they take turns or carry an assigned role under different interests. We instantiate four tasks as 140 scenarios and record 7,560 conversations, covering six self- and cross-play pairings. Throughout the experiments, we observe that how a model behaves continually reshapes its partner. We thus demonstrate that each model must be both the examiner and examinee of the other, and no single fixed interlocutor can play both parts. We will release the scenarios, role prompts, and recording protocols between two full-duplex models, without any pre-recorded audio.

1. Introduction

Existing full-duplex benchmarks use fixed or agenda-bound interlocutors, even though conversational behavior emerges from two coupled speakers. DyaFDB evaluates two role-assigned models directly and grades both sides, probing how partner and task conditions shape behavior.

  • Prior benchmarks use recorded or teacher-forced input, or reactive examiners that administer tests but are not graded; their fixed interlocutors cannot capture both speakers' joint influence.Source: §1 · p. 2, Table 1 · p. 2
  • Spoken interaction makes dyadic evaluation especially relevant because interruption, silence, and latency can influence a partner while the model commits outputs continuously.Source: §1 · p. 2
  • DyaFDB pairs two free-running models under assigned roles, scores both offline, and varies partner model and role across four cooperative or conflicting tasks.Source: §1 · pp. 2–3
  • The benchmark pairs three open models in self- and cross-play, records 7,560 conversations, and will release scenarios, role prompts, and recording protocols without fixed audio.Source: §1 · pp. 2–3
  • Performance compounds partner behavior, task structure, and disposition to engage; the same defender leaks its secret in 39–63% of conversations depending on the attacker.Source: §1 · p. 2

2. Related Work

Prior work advances full-duplex speech generation and measures conversational behaviors, but spoken dialogue benchmarks generally keep the interlocutor fixed or agenda-bound and grade one side. DyaFDB draws on interactive evaluation and dialogue task designs to measure both participants in live dyads.

  • Full-duplex modeling enables simultaneous listening and speaking, following cascaded and half-duplex systems; dGSLM modeled both conversational channels, and Moshi popularized synchronized streams.Source: §2 · p. 3
  • Existing full-duplex benchmarks use fixed interlocutors or agenda-bound simulated users and grade one side, so they cannot separate model decisions from partner-induced ones.Source: §2 · p. 3
  • Interactive evaluation in games, multi-agent learning, robotics, and text dialogue uses populations, cross-play, or pairwise interaction rather than a single fixed opponent.Source: §2 · p. 3
  • DyaFDB measures backchanneling, barge-in, and turn-takeover rates in every task, extending turn-taking measures previously used with fixed partners.Source: §2 · p. 4
  • Prior dialogue studies include negotiation, persuasion, information pooling, and withholding tasks; DyaFDB organizes its suite into cooperative Coordination and incompatible-goal Conflict tasks.Source: §2 · p. 4, Table 2 · p. 4

3. DyaFDB: Dyadic Evaluation Framework for Full-Duplex Models

DyaFDB pairs full-duplex models directly under cooperative or conflicting roles, then scores both from recorded conversations. Its dyadic results show that turn-taking and most task scores depend on the partner or model-partner combination, not just the model.

3.1. Benchmark Design

  • Models exchange audio on a lockstep clock, with each hearing its partner’s output one 80 ms frame later; MiniCPM-o uses an adapter for its chunk-based input.The bridge runs at 24 kHz; the adapter buffers frames into 16 kHz chunks.Source: §3.1 · p. 4
  • DyaFDB groups four tasks into Coordination, with compatible goals, and Conflict, with incompatible goals; scenarios assign each model a private role prompt within a shared situation.The tasks test role adherence, pooling split information, contesting the channel, and pursuing opposed goals.Source: §3.1 · pp. 4–5

3.2. Turn-Taking Metrics in a Live Dyad

  • TORpause is mainly model-decided (η²_model = 0.74), while TORend is partner-decided (η²_partner = 0.65) and barge-in success similarly relies on the partner yielding.Figure 2 decomposes turn-taking metrics by model, partner, and their interaction.Source: Figure 2 · p. 5, §3.2 · p. 5
  • Role adherence is the only task score mostly decided by the model; every other score is a property of the pair, including symmetric and conflicting tasks.Figure 2 presents the task-score variance decomposition.Source: Figure 2 · p. 5, §3.2 · p. 6

3.1. Benchmark Design

  • Evaluation uses six self- and cross-play pairings, recording 7,560 one-minute conversations and scoring both turn-taking and task criteria offline from audio and text channels.An LLM judge and mechanical measures are used, and scores are reported by pairing and as per-model averages.Source: §3.1 · p. 5, §3.2 · p. 5

3.2. Turn-Taking Metrics in a Live Dyad

  • Backchanneling has the largest interaction term (η²_model×partner = 0.32): PersonaPlex backchannels at 4.6 against Raon, while MiniCPM-o is at 0.1 in self-play.These behaviors appear only for particular model combinations, highlighting the relevance of partner stance and attitude.Source: §3.2 · pp. 5–6

4. Task Scores of DyaFDB

Coordination results vary with the partner: role adherence shifts across archetypes, and information pooling succeeds infrequently. In conflict tasks, outcomes depend on the model’s role and opponent, with Raon strong in coordination but weak at holding the floor or defending a secret.

4.1. Coordination: Role Adherence and Information Pooling

  • Role-adherence rankings are Raon above MiniCPM-o above PersonaPlex in both archetypes, but a digressing partner lowers PersonaPlex’s score and raises Raon’s.PersonaPlex changes from 0.51 to 0.29, MiniCPM-o from 0.62 to 0.59, and Raon from 0.68 to 0.83.Source: §4.1 · p. 6
  • Information-pooling teams including Raon solve 23–35% of runs; teams without Raon score below 20%, while models disclose only 39% of their reasons.About half the pairs reach no decision within a minute.Source: §4.1 · p. 7

4.2. Conflict: Channel and Goal

  • Raon takes more channel occupancy than other models in both contested-channel archetypes, while each model’s occupancy barely changes between competing and avoiding partners.Raon reaches 0.75 against MiniCPM-o and 0.62 against PersonaPlex; as an avoider, MiniCPM-o yields more than Raon.Source: §4.2 · p. 8
  • Preference choices are reached in only 38% of runs; Raon leaks secrets most often (48% on average), while PersonaPlex extracts them most often (46% on average).More than 80% of runs that reach a decision favor Raon; MiniCPM-o has the lowest average leak rate at 29%.Source: §4.2 · p. 8
  • Raon is weakest in both holding roles—avoiding talk and defending a secret—and the same defender’s leak rate ranges from 39% to 63% depending on the attacker.This range shows how a fixed-interlocutor benchmark can report an opponent-dependent result.Source: §4.2 · p. 8

5. Cross-Task Analysis

Across tasks, low scores can reflect failure to act as well as failure after acting, and longer windows only partly address missing actions. Outcomes also depend on the game’s role structure and the models playing each role.

5.1. Success Depends on Action

  • The most accurate decision maker and best extractor act least, while action rates and success given action rank models differently.MiniCPM-o has 𝑝(score | decided) = 0.40 but 𝑝(decided) = 0.41 in T2; PersonaPlex asks least in both Conflict tasks yet succeeds most once it asks.Source: §5.1 · p. 8, §5.3 · p. 10

5.2. Conversation Time Window

  • 30-to-60-second windows raise every measure, but scores largely saturate beyond 60 seconds and longer conversations rarely change how models attempt.At 120 seconds, Raon’s T3-2 ask rate rises from 0.35 to 0.50 and agreement from 0.14 to 0.29, while T2’s higher delivery rate does not yield a higher score.Source: §5.2 · pp. 8–9, Figure 5 · p. 9

5.3. Game Balance

  • Game structure favors avoiders and defenders in asymmetric roles, while the money games favor the side receiving money; model differences can exceed role advantage.Raon wins 77% as claimant while PersonaPlex wins 32%; Figure 6 distinguishes all valid runs from balances conditional on a deal.Source: §5.3 · p. 9, Figure 6 · p. 10

6. Conclusion

DyaFDB evaluates full-duplex dialogue models in live dyads, with both participants scored. Across its four Coordination and Conflict tasks, performance depends on partner behavior and task structure as well as a model’s disposition to engage.

6. Conclusion

  • Across four Coordination and Conflict tasks, turn-taking metrics and task scores are properties of the pair, shaped by partner behavior, task structure, and engagement.The authors recommend testing duplex agents across diverse opponents before trusting them with private information.Source: §6 · p. 10

Ethics Statement

The paper frames DyaFDB’s attack task as a test of disclosure under pressure, while leaving behavioral valuation to deployment contexts. Its local-model infrastructure uses a shared logical clock, and Raon speech/text divergence can contaminate some sessions.

Ethics Statement

  • The attack archetype uses fictional secrets to test disclosure under social pressure, but the benchmark measures behavior rather than whether that behavior is desirable.The authors leave valuation of pressing, extracting, or conceding to the deployment context.Source: Ethics Statement · p. 10

A. Infrastructure and Corpus

  • A shared logical clock advances both local models in concurrent 80 ms steps, avoiding network jitter and allowing runs to execute slower or faster than real time.Adapters translate each model’s native interface; MiniCPM-o’s asynchronous worker can make its pairings run at up to ∼1.3× wall-clock time.Source: A. Infrastructure and Corpus · p. 15
  • Sessions end after 60 seconds or when a model finishes, and recordings pair stereo audio with each model’s text channel without ASR in the scoring path.The opener is counterbalanced and gets a 1.5-second lead-in; models use pinned voices, and Raon exposes no seed parameter.Source: A. Infrastructure and Corpus · p. 15
  • In a small fraction of Raon sessions, emitted speech diverges from its text stream, contaminating runs because partners respond to audio absent from the transcript.The authors attribute the divergence to speech-token decoding and report that it affects roughly 5% of Raon sessions; flagged sessions are re-recorded and checked.Source: A. Infrastructure and Corpus · pp. 15–16

B. Task and Scenario Design

DyaFDB builds scenarios from a shared situation and two private role prompts, then instantiates coordination and conflict tasks with distinct partner stances or goals. Role assignment and speaking order are crossed to separate those factors from model effects.

B. Task and Scenario Design

  • Each scenario combines a shared situation with two private role prompts, and crossed role assignment and speaking order counterbalance those factors.Each archetype is instantiated in 20 scenarios.Source: B. Task and Scenario Design · p. 16

T1 Role adherence

  • T1 tests whether a model holds its assigned role with an engaging or digressing partner, using complementary roles or an unrelated-topic stance.Examples include a doctor and patient, or a cook whose partner repeatedly returns to a story about a dog.Source: B. Task and Scenario Design · p. 16, T1 Role adherence · p. 17

T2 Information pooling

T3 Contested channel

  • T3 contests the shared speech channel through competing promotion briefs or a presser seeking permission from an avoider.The competing-brief roles each must communicate three product points; the permission-seeking presser must keep asking while the avoider answers briefly and avoids commitment.Source: B. Task and Scenario Design · p. 16, T3 Contested channel · p. 17

T4 Contested goal

  • T4 uses contested goals including preference, disclosure under attack, compensation claims, and price negotiation.The examples include a shared venue choice; the negotiation archetype settles on one agreed number, while no-deal loses for both sides.Source: B. Task and Scenario Design · p. 16, T4 Contested goal · p. 17

C. Turn-Taking Metrics and Variance Decomposition

The section adapts turn-taking measures to live dyads and decomposes variation across task, model, and partner. It also specifies how the external judge scores conversations and validates those judgments against another judge and human experts.

C. Turn-Taking Metrics and Variance Decomposition

  • Turn-taking measures adapt pause handling, turn-taking, backchannel, and interruption to two live speakers, with no per-event correct behavior.Barge-in success scores whether the partner yields; backchannels must leave the partner speaking 0.5 seconds after the short utterance ends.Source: C. Turn-Taking Metrics and Variance… · p. 18
  • A fixed-partner benchmark cannot separate model, partner, and pair-specific contributions to differences across model-partner cells.The decomposition measures how much variance across cells each source explains.Source: C. Turn-Taking Metrics and Variance… · p. 18
  • The task term explains at most 0.02 of turn-taking variance, while per-task decompositions split variance across model, partner, and their interaction.For each task-score matrix, the three terms sum to 1; the small task term motivates equal weighting across the seven tasks.Source: C. Turn-Taking Metrics and Variance… · p. 19, Table 6 · p. 19

D. Judge Rubric and Validation

  • The LLM judge scores anonymized, time-ordered transcripts outside the recording loop, while mechanical measures bypass the judge.For role timing, opening small talk does not count as entering the role; for T2 and T4-1, the judge identifies the settled option for mechanical comparison with candidates.Source: D. Judge Rubric and Validation · p. 19
  • The two LLM judges differ by 0.04 on average across 36 judged cells, yet preserve every model ordering.Opus-5 agrees with the human majority on 89.3% of sampled seats and Sonnet-5 on 85.0%; expert-expert agreement is 77–88%.Source: D. Judge Rubric and Validation · p. 19, Table 8 · p. 20

E. Comparison with Text LLMs

The text-pair comparison finds that role adherence is relatively close, but interactive tasks expose larger gaps between text and speech models. Behavioral profiles also differ by model, with no absolute winner across roles and tasks.

E. Comparison with Text LLMs

  • Text models ask in 68–90% of T3-2 runs versus 16–39% for speech models, and succeed more often once they ask.In T2, the text pair decides as often as Raon but is more often correct once it decides; Qwen3-8B discloses 74% of its reasons versus 34–47% for speech models.Source: E. Comparison with Text LLMs · p. 20, Table 9 · p. 21
  • Role adherence is the closest text-speech comparison, with Raon 0.20 below its own backbone LLM.The comparison uses two text models on the same role-prompted scenarios, with a 240-word budget and a 20-turn cap; three tasks are excluded.Source: E. Comparison with Text LLMs · p. 20, Table 9 · p. 21

F. Per-Model Behavioral Profiles

  • There is no absolute winner: PersonaPlex engages slowly but performs effectively once engaged, MiniCPM-o is careful, and Raon commits readily but can disclose what it should hold.Profiles average each model over its three pairings; whether a disposition helps depends on the assigned role.Source: F. Per-Model Behavioral Profiles · p. 21, Table 10 · p. 21

G. Additional Tasks: Claim and Negotiation Games

The claim and negotiation results vary by model and role, and the full matrix reports receiving-side scores among runs that reached a deal. Negotiation outcomes often reflect which side held its opening price.

G. Additional Tasks: Claim and Negotiation Games

H. Additional Model: Lychee-FD

Lychee-FD extends the experiments to one self-play and three cross-play pairings, but its recordings include substantially more audio garbles than Raon’s. Its behavior combines strong floor-taking with little disclosure, producing contrasting performance as a defender and teammate.

H. Additional Model: Lychee-FD

  • Lychee-FD was recorded in one self-play and three cross-play pairings, and its audio garbles—especially in self-play—were more frequent than Raon’s.Recordings still garbled after five re-recordings count as failures; they are retained in T3, where occupancy is defined on any audio.Source: H. Additional Model: Lychee-FD · p. 22
  • Lychee-FD takes the floor from PersonaPlex and MiniCPM-o like Raon, but discloses little, making it the best defender and weakest teammate.It also has the fewest preference wins and lowest pooling scores.Source: H. Additional Model: Lychee-FD · p. 22
  • As a partner, Lychee-FD lowers MiniCPM-o and Raon role adherence by 0.06 beside an engaging partner.The extended results preserve the overall trend across metrics; T4-2 leak variance is defender-dominated when the strongest defender is in the column.Source: H. Additional Model: Lychee-FD · pp. 22–23

I. Limitations

The discussion identifies benchmark-side and model-side limitations, then outlines directions for future work.

Benchmark limitations

  • Per-model averages depend on the measured partners, so the authors say ratings over pairings will be needed as the model pool grows.Source: §I · p. 23
  • Current turn-taking and task scores do not distinguish natural conversation from role-holding or goal achievement through ignoring, steering, enforcing, or persuading.Source: §I · p. 23
  • Human raters can distinguish these conversational approaches, but rating every run by hand is slow and mostly subjective.Source: §I · p. 23
  • All study partners are models; adding real humans would require a large participant pool, which the authors leave to future work.Source: §I · p. 23

Model limitations

  • Current models struggle to withhold roles and to treat speaking as action; longer windows rarely change attempts, while behavior becomes less consistent across rounds.Source: §I · p. 23

Future work

  • The authors propose treating deciding, asking, and disclosing as a social skill, training against diverse partners, and making restraint as instructable as engagement.Source: §I · p. 23
Loading 2610.08125v1…