PaperlayerView exact revision on arXiv

Source-linked AI summary

From Metrics to Natural Dialogue: French Full-Duplex Benchmark for Spoken Dialogue Models

Hamid Soltani, Gilles Boulianne

arXiv:2609.10765v1eess.AS

TL;DR

The paper investigates whether full-duplex benchmark metrics behave differently when model and benchmark languages do not match. It introduces two French benchmarks from spoken resources and evaluates human–human conversations with FDB metrics. Timing-related measures are relatively stable across languages, but content-dependent evaluation is more sensitive to mismatch, and metric optimization can trade off with naturalness.

  • Problem

    It is unclear how full-duplex benchmark metrics behave when a model is evaluated in a non-target language, despite evaluating timing-sensitive dialogue skills as well as response quality.

    Source: §1 · p. 2

  • Method

    The paper introduces CALLFC-FDB and MEDIA-FDB from Canadian and European French spoken resources and evaluates human–human conversations using FDB metrics.

    Source: §1 · p. 2

  • Results

    Timing-related behaviors remain relatively stable across languages, while content-dependent user-interruption ratings degrade when model and benchmark languages do not match.

    Source: §5 · p. 7

  • Takeaways & Limitations

    Human–human conversations do not always yield optimal FDB scores, so benchmark performance alone may not reflect human-like interaction.

    Source: §5 · p. 7

    The proposed French benchmarks adapt only FDB v1 tasks; extending them to other FDB tasks remains future work.

    Source: Limitations · p. 7

Abstract

from arXiv · show

Full-duplex spoken dialogue models aim to make voice agents more natural by allowing them to listen, speak, pause, and respond during ongoing conversation. However, it is not clear whether full-duplex benchmarks behave the same way when models are evaluated in a different language. To investigate this, we introduce a French full-duplex benchmark (FDB) with two variants, CALLFC-FDB for Canadian French and MEDIA-FDB for European French, and compare them with an English FDB. Built from real spoken resources, these benchmarks evaluate key full-duplex skills, including pause handling, turn-taking, backchannels, and interruptions. Beyond introducing French FDBs, we evaluate human--human conversations with FDB metrics to better understand the values these metrics take in real-world dialogue. Our analysis reveals that most timing-based metrics behave similarly across languages, while content-based evaluation degrades under language mismatch. We also find a trade-off between optimizing benchmark metrics and preserving conversational naturalness.

1 Introduction

The paper asks whether full-duplex benchmark metrics behave consistently across evaluation languages and introduces two French benchmarks alongside human–human metric comparisons. It reports that timing-based metrics are broadly consistent across languages, while content-based evaluation is more sensitive to language mismatch.

1 Introduction

  • Full-duplex systems listen and speak simultaneously, supporting turn-taking, backchannels, and overlapping speech unlike half-duplex systems that wait for the user’s turn to end.Cascaded systems can add latency and lose acoustic information during speech-to-text and text-to-speech conversion, while speech-to-speech systems preserve conversational information needed for natural interaction.Source: §1 · pp. 1–2
  • The open question is whether FDB metrics behave differently when a model is evaluated in a non-target language, especially for timing-sensitive dialogue behaviors.Dedicated French full-duplex benchmarks were missing, and French interactional cues such as euh and d’accord do not map directly onto English translations.Source: §1 · p. 2
  • Most timing-based metrics remain broadly consistent across languages, while content-based evaluation is more sensitive to language mismatch.The paper also reports a trade-off between optimizing FDB metrics and maintaining natural spoken interaction.Source: §1 · p. 2
  • CALLFC-FDB and MEDIA-FDB bring real-world Canadian and European French conversations into full-duplex evaluation.The benchmarks support evaluation across different French varieties and interaction settings.Source: §1 · p. 2
  • Human–human conversations provide reference values for FDB metrics, and the analysis finds that metric optimization can trade off against conversational naturalness.Real dialogue is more flexible than fixed metric targets.Source: §1 · p. 2

2 French Full-Duplex-Bench

The French benchmarks adapt the four core FDB v1.0 tasks, while task availability and expected language dependence differ across tasks.

2.1 FDB Tasks

  • Pause handling, backchanneling, smooth turn-taking, and user interruption are the four adapted FDB v1.0 tasks.They assess silence during an ongoing user turn, listener feedback without taking the turn, responding after the user finishes, and handling interruptions that introduce a new intent.Source: §2.1 · pp. 2–3
  • MEDIA-FDB omits backchanneling because the corpus lacked enough valid examples of system backchannels.Source: §2.1 · p. 3
  • The authors expect pause handling to be weakly language-dependent and user interruption to be the most language-dependent task.Turn-taking may depend on language matching, while backchanneling also involves selecting appropriate feedback and its timing and duration.Source: §2.1 · p. 3

2.2 FDB Metrics

FDB metrics largely assess voice activity and timing, with user-interruption response quality as a content-dependent exception.

2.2 FDB Metrics

  • Takeover Rate scores silence and backchannels as 0 and other responses as 1, with preferred direction varying by task.Lower TOR is preferred for pause handling and backchanneling; higher TOR is preferred for smooth turn-taking and user interruption.Source: §2.2 · p. 3
  • Backchannel Frequency counts backchannels per duration, while Jensen-Shannon Divergence compares model and ground-truth backchannel timing distributions.Lower JSD means greater similarity in backchannel locations and durations; smooth turn-taking and user-interruption latency measure the time from the user’s turn end or interruption to the model response.Source: §2.2 · p. 3
  • The user-interruption Rating is an LLM-based response-quality score on a 0-to-5 scale.Source: §2.2 · p. 3
  • Most FDB metrics use voice activity, while the user-interruption rating depends on response content and can penalize a language mismatch despite appropriate timing.Latency can also reflect system architecture and implementation, especially in cascaded pipelines.Source: §2.2 · p. 3

2.3 CALLFRIEND and MEDIA Data

CALLFC-FDB and MEDIA-FDB draw on two French spoken-dialogue corpora, covering informal conversation and structured task-oriented dialogue.

2.3.1 CALLFRIEND French Canadian

  • CALLFRIEND French Canadian contributes 26 hours of unscripted telephone conversations between native Canadian French speakers.Recorded in 1996, the conversations were informal, with participants often calling family or friends and discussing any topic.Source: §2.3.1 · p. 3

2.3.2 MEDIA

  • MEDIA is a French human–machine dialogue corpus for hotel reservations and tourist information, collected using a Wizard-of-Oz setup.Users believed they were interacting with an intelligent system, while a human operator produced the responses.Source: §2.3.2 · p. 3

2.3 CALLFRIEND and MEDIA Data

  • Together, CALLFRIEND and MEDIA cover informal open-domain and structured task-oriented dialogue.Source: §2.3.2 · p. 3

2.4 French FDB Dataset Preparation

The preparation converts continuous French conversations into FDB-compatible samples, using corpus-specific transcription and turn extraction alongside shared event selection, validation, and export stages.

2.4 French FDB Dataset Preparation

  • CALLFC-FDB and MEDIA-FDB share output, scoring, materialization, and export stages, but differ in transcript preparation and turn extraction.CALLFC-FDB additionally recovers speaker information from overlap; MEDIA-FDB uses existing speaker assignments and overlap annotations.Source: §2.4 · p. 4, §2.4.1 · p. 4

2.4.1 Transcription Process

  • CALLFC-FDB merges manual transcripts with automatic channel transcripts to recover overlap words, while MEDIA-FDB aligns its manual transcripts to audio for word timings.The merged CALLFC-FDB transcript adds 44,165 overlap words—about 11% of its final 386K words.Source: §2.4.1 · p. 4

2.4.2 Turn and sentence extraction

  • MEDIA-FDB takes turns from manual transcription; CALLFC-FDB groups words separated by no more than 2.0 seconds, and processes each speaker as speaker and WoZ in turn.A new sentence starts at an inter-word gap greater than 0.2 seconds.Source: §2.4.2 · p. 4

2.4.3 FDB event extraction

  • Event selection extracts pause, backchannel, smooth-turn-taking, and interruption examples from paired speaker turns.Pauses exceed 0.7 seconds; backchannels come from turns of at least 2.0 seconds; smooth turns have under 1.0 second between turns; interruptions begin before the prior WoZ turn ends and are not backchannels.Source: §2.4.3 · pp. 4–5

2.4.4 Scoring, validation, and export

  • 41.4% of MEDIA and 26.3% of CALLFC candidates remained in the final datasets after quality ranking and the 150-sample-per-task cap where sufficient candidates existed.Validation filters remove unsuitable examples, then candidates are ranked for clarity and task representation before export.Source: §2.4.4 · p. 5

3 Evaluation

The evaluation compares English and French spoken dialogue systems on English and French FDBs, using human conversations as a reference and adapting the evaluation pipeline for French.

3 Evaluation

  • English reference models Freeze-Omni and PersonaPlex-7B-v1 are evaluated alongside English and French cascaded systems on the English and French benchmarks.The English and French cascades use the same STT–LLM–TTS structure, with language-specific Vosk and Piper models and Qwen3:8b.Source: §3 · p. 5
  • Human–human baselines provide a natural reference, while Table 1 reports extracted French sample counts by task and dataset.The English and French cascade latency is expected to be relatively high because of the simple cascaded design; the authors attribute most delay to LLM generation.Source: §3 · p. 5, Table 1 · p. 5
  • The evaluation replaces Parakeet-TDT v2 with v3 because v2 does not support French and v3 supports multilingual ASR.For user-interruption ratings, the evaluation uses llama3.3:70b as an LLM judge with the official FDB prompt, rubric, and generation settings.Source: §3 · pp. 5–6

4 Results

Across the evaluated systems and benchmarks, most timing-based metrics remain broadly comparable under language mismatch, while content-based user-interruption ratings are strongly language-dependent.

4 Results

  • For pause handling, English FDB generally yields lower or comparable TOR values across both English and French models.The authors suggest this behavior mainly depends on voice activity, such as silence within a turn, rather than the evaluated model's language.Source: §4 · p. 6
  • PersonaPlex has the highest backchannel frequency and lowest JSD on English FDB and CALLFC-FDB, while smooth-turn-taking TOR improves for cascades when benchmark and model languages match.Freeze-Omni remains stable on smooth turn-taking, whereas PersonaPlex drops on the French benchmarks.Source: §4 · p. 6
  • Most voice-activity-based metrics remain broadly comparable across language mismatch, but user-interruption ratings drop sharply when model and benchmark languages differ.Language-matched settings receive much higher ratings.Source: §4 · p. 6
  • The English cascade has lower latency than the French cascade on smooth turn-taking and user interruption, while human conversations do not always yield the highest interruption rating.On MEDIA-FDB, the French cascade receives a higher interruption rating than humans; human interruption latency is also higher than for noncascaded English models.Source: §4 · pp. 6–7

5 Conclusion

The paper introduces two French FDB datasets and finds that language mismatch affects full-duplex skills differently, while human benchmark scores do not always reflect natural interaction.

5 Conclusion

  • CALLFC-FDB and MEDIA-FDB provide FDB-compatible benchmarks built from real-world Canadian and European French conversations.Timing-related behaviors remain relatively stable across languages, while user-interruption ratings degrade when model and benchmark languages do not match.Source: §5 · p. 7
  • Evaluation language, system architecture, and latency should be considered together when selecting and evaluating full-duplex systems.The conclusion distinguishes acoustic-timing competence from language-dependent interaction competence.Source: §5 · p. 7
  • Human–human conversations do not always yield optimal FDB scores, and future work will test additional FDB tasks and overlapping-speech scenarios.The paper identifies corpus-level differences as a limitation of the language comparison; closely matched multilingual datasets could better isolate language effects.Source: §5 · p. 7

Limitations

The French benchmarks cover FDB v1, while extending them to later tasks and scenarios remains future work. The comparison is also constrained by system and corpus differences, and FDB metrics do not fully capture several dimensions of conversational quality.

Limitations

  • The French benchmark adaptation covers FDB v1; future work should extend it to FDB v1.5 and FDB v2, including overlapping speech and multi-turn scenarios.The study also limits its language-mismatch evaluation to FDB v1.Source: Limitations · p. 7
  • The latency comparison lacks a fast French speech-to-speech model, and more comparable French and multilingual systems are needed to isolate language from architecture.Source: Limitations · p. 7
  • The language comparison varies in corpus, domain, and conversational formality, so closely matched multilingual datasets could better isolate language effects.Source: Limitations · p. 7
  • FDB metrics mainly assess timing and turn-taking, but do not fully capture conversational naturalness, user satisfaction, or long-term dialogue coherence.The passage recommends interpreting benchmark scores alongside qualitative analysis and human-centered evaluation.Source: Limitations · p. 7

Ethics Statement

The release includes annotations and preparation scripts, not audio; users must obtain the source datasets from their distributors and follow their licenses.

Ethics Statement

  • The authors release annotations and preparation scripts, but not original or extracted audio.Source: Ethics Statement · p. 7
  • Users must obtain CALLFRIEND and MEDIA from their official distributors and follow the corresponding dataset licenses.Locally generated samples remain subject to the original dataset licenses.Source: Ethics Statement · p. 7
  • The authors withhold speaker metadata and personal identifiers, limiting additional privacy risks.Source: Ethics Statement · p. 7

A Confidence Intervals

Table 3 reports 95% confidence intervals for all results in Table 2.

A Confidence Intervals

Loading 2609.10765v1…