PaperlayerView exact revision on arXiv

Source-linked AI summary

Prompts versus Rules: Auditing and Controlling Speech Naturalness Behaviors in Voice User Simulators

Riqiang Wang, Elena Khasanova, Harsh Saini, Lex Konnelly, Parsa Kavehzadeh, Matthias Lee, Mohamed Attia

arXiv:2610.11015v1cs.CL

TL;DR

Voice-agent evaluation depends on user simulators, yet prior work has often not examined whether configured naturalness controls are realized. This paper audits prompted and rule-based controls for disfluency, interruption, and backchanneling, finding that prompting is unreliable while rule-based injection offers more controllable behavior. It argues that evaluations should report realized behaviors against opportunity baselines rather than rely on settings alone.

  • Problem

    The study addresses limited examination of whether configured disfluency, interruption, and backchannel controls are realized in user-simulator speech, even though the simulator is part of the evaluation measurement.

    Source: §1 · pp. 1–2

  • Method

    The paper audits τ-Voice and prompting across three models, compares them with rule-based naturalness controls, and measures realized behavior including rates per qualifying opportunity.

    Source: §1 · p. 2, §3.2 · p. 4, §4 · p. 5

  • Results

    Prompted controls show operational shortfalls, while rule-based injection provides monotonic controllability, realistic disfluency distribution, and natural over-dispersed turn-to-turn variance while eliminating latency and extra model invocation costs.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Evaluation harnesses should audit and report realized naturalness behaviors relative to opportunity baselines rather than rely on unverified configurations or agent-side metrics alone.

    Source: §6 · p. 8

    The evaluation covers three models in a cascaded simulator architecture with specific settings and one fixed agent, so the observed effects may not hold under substantially different conditions.

    Source: Limitations · p. 9

Abstract

from arXiv · show

As voice agents gain more popularity commercially, the user simulators used to evaluate the deployed agents are also being developed to include more realistic, variable, and diverse speech naturalness behaviors -- disfluency, interruption and backchanneling. The quality of the user simulator directly affects the validity of agent evaluation results. However, we find that most studies so far have not examined in detail whether the intended configuration for these behaviors is realized in the simulation. In this study, we audit the realized naturalness behaviors of tau-Voice, our own LLM-based prompting approach across three models, and our rule-based injection algorithm for disfluency, interruption, and backchanneling. We find that prompting for these behaviors is unreliable and produces speech inconsistent with the instructions, placed and distributed less naturally than the instruction implies. In contrast, our rule-based, model-free algorithm produces controllable and diverse naturalness behaviors more aligned with natural speech. Our results suggest that LLMs not purpose-trained for user simulation are not sufficient on their own to represent authentic user behavior, and that linguistically informed deterministic approaches or specialized models are needed to close the gap; auditing and reporting realized naturalness behaviors, rather than configured settings, is what makes that gap visible.

1 Introduction

Voice-agent evaluation increasingly uses speech-to-speech LLM user simulators, but the simulator itself becomes part of the measurement. This study audits whether configured controls for disfluency, interruption, and backchanneling produce the intended behavior, comparing prompting with rule-based controls.

1 Introduction

  • Speech conversation makes configuration-to-behavior gaps important because natural turn-taking includes disfluency, overlapping speech, backchannels, and interruptions.The paper calls these three phenomena speech naturalness behaviors and asks whether controls yield speech consistent with their settings.Source: §1 · p. 2
  • Prior harnesses publish user-simulator configurations but do not report the naturalness behaviors those settings actually produce.The cited examples report agent-side interruption or filler measures rather than realized behavior on the simulator side.Source: §1 · p. 2
  • The study audits realized disfluency, interruption, and backchanneling rates from τ-Voice and compares them with deterministic rule-based controls.Disfluency is measured per 1,000 words; interruption and backchanneling are measured per qualifying opportunity and per agent turn.Source: §1 · p. 2
  • Controllability is distinct from realism, but behavioral diversity is described as a necessary condition for realism and robust agent evaluation.Source: §1 · p. 2
  • Rule-based controls achieve monotonically controllable interruption and backchannel rates, wider disfluency diversity, and eliminate model latency and API cost.The paper also presents an empirical audit over live full-duplex telephony transport.Source: §1 · p. 2

2 Related work

Prior spoken-user simulators and voice-agent harnesses address parts of spoken interaction, but often lack reproducibility or user-side naturalness auditing. This work evaluates a live, cascaded simulator and introduces event-driven rule-based controls.

2 Related work

  • Spoken user simulators include scripted speech, prompted disfluency, and task-oriented dialogue with slot disclosure, prosody, and barge-in, but SpokenUS weights and runtimes are unavailable for external execution.Source: §2 · p. 2
  • τ-Voice uses a fixed 2.0-second polling schedule with dedicated LLM calls, while relying on text prompts for disfluency.Its prompts contain conflicting filler instructions and it reports no realized disfluency rates.Source: §2 · pp. 2–3
  • Voice evaluation harnesses commonly measure agent-side outcomes without auditing the naturalness behaviors realized by the simulated user.Examples include agent-only filler measurement, non-reactive user recordings, pre-generated audio, and disabled or unmonitored controls.Source: §2 · p. 3
  • Native audio models and prompting approaches offer naturalness-related capabilities, but cited systems have constraints including self-dialogue, half-duplex operation, or weak instructed-versus-generated correlations.The reported correlation range is r = 0.25–0.54; the paper also reports rapid saturation and turn-initial concentration for prompted disfluencies.Source: §2 · p. 3
  • The work evaluates a real-time cascaded simulator against deployed commercial agents over live telephony and uses event-driven rules for naturalness controls.The controls operate at mid-turn pause boundaries and are presented as an alternative to unverified prompts.Source: §2 · p. 3

3 Experimental setup

The simulator interacts with a black-box voice agent over full-duplex audio and uses speech activity events to trigger turn-taking behavior. Rule-based controls separately govern disfluency, interruption, and backchanneling through configurable tendency parameters.

3.1 User simulator design

  • The simulator and target agent exchange audio over telephony in a black-box setup, with no access to the agent’s internal transcripts, weights, prompts, or generation probabilities.The simulator is a cascaded STT–LLM–TTS pipeline with pre-synthesis hooks for deterministic disfluency injection; its default models are from the gpt-4o-mini family.Source: §3.1 · p. 3
  • Voice activity events trigger decisions at 200 ms mid-speech pauses, while 500 ms turn-end silence triggers standard response generation.Pause-triggered decisions avoid continuous polling; turn handling requires at least 500 ms plus model latency, versus human response gaps of 110–130 ms.Source: §3.1 · pp. 3–4

3.2 Rule-Based Naturalness Controls

  • Interruption and backchannel tendencies β and γ govern decisions at 200 ms pauses; interruption is checked first, and backchannels use pre-synthesized listener tokens if it does not trigger.At tendency 0.0, a behavior is disabled.Source: §3.2 · pp. 4–5, Figure 2
  • Rule-based disfluencies are rewritten into generated text before speech synthesis according to tendency α, with the same utterance text producing reproducible placements.The rewrite is post-generation to avoid changing future conversational context.Source: §3.2 · p. 4
  • The disfluency algorithm excludes boundaries adjacent to digit- or @-containing words and tunes filler, repetition, and pause-token distributions separately.The study does not discuss pause-token realization because it depends on speech synthesis.Source: §3.2 · p. 5

4 Methodology

The methodology evaluates prompted and injected disfluency on replayed dialogue positions, then measures interruption and backchanneling during live full-duplex interaction. For the latter, rates are normalized by qualifying opportunities as well as agent turns.

4 Methodology

  • The disfluency comparison uses 299 paired dialogue positions with identical contexts and histories across conditions.Positions were sampled from 858 user response positions drawn from 131 successful conversations; one position was excluded across all conditions.Source: §4 · p. 5
  • Disfluency experiments compare prompted and rule-based injection across three language models and model-independent injection settings.The study uses six prompt conditions per model—gpt-4o-mini, gpt-5, and Gemini 3 Pro Preview—and four injection settings.Source: §4 · p. 5
  • Live interruption and backchannel rates use a 200 ms mid-speech pause as the opportunity denominator, excluding turn-end silences and periods when the simulator is speaking.Rates are reported per opportunity and per agent turn; τ-Voice’s fixed 2.0-second ticks trigger classifier consultations every ten ticks.Source: §4 · pp. 5–6
  • Prompted and injected disfluencies are evaluated on replayed turns against the deployed agent, with τ-Voice sampled over 20 sessions and seven paired injection conditions across 20 sessions each.Source: §4 · p. 6

5 Results and Analysis

Prompting does not reliably realize the requested speech behaviors: disfluencies are underdelivered and poorly distributed, while τ-Voice interrupts aggressively and rarely backchannels. Rule-based controls provide more predictable, monotonic rates across behaviors, though realized behavior depends on the target agent’s speech cadence.

5.1 Disfluency: Prompting versus Rule-Based Injection

  • Prompted filler rates underdelivered and nearly plateaued at high requested frequencies, while rule-based injection scaled close to linearly for fillers.Baseline instruction-tuned models produced virtually zero fillers without explicit disfluency instructions.Source: §5.1 · p. 6, Table 1 · p. 7
  • Prompting produced almost exclusively fillers, with repetitions reaching only 2.96 per 1,000 words on gpt-4o-mini versus 47.06 under rule-based injection at α = 1.00.Prompted outputs contained no self-initiated repairs, which were not part of the injection design.Source: §5.1 · p. 6, Table 1 · p. 7
  • Turn-initial fillers made up 62.8%–67.3% of gpt-4o-mini prompted fillers, versus 11.1%–15.6% across operational injection settings.Injection also placed disfluencies at clause boundaries and internal lexical sites; its turn-initial share was non-monotonic as α increased.Source: §5.1 · p. 6, Table 1 · p. 7

5.2 Interruption and Backchanneling

  • Interruption tendency β scaled realized interruptions from 0.235 to 0.538 per agent turn, while backchannel tendency γ scaled backchannels from 0.545 to 0.844 per turn.Both behaviors scaled monotonically, although the mapping from configured settings to realized rates was non-linear and depended on the target agent’s speech distribution.Source: Table 2 · p. 7, §5.2 · p. 8

5.1 Disfluency: Prompting versus Rule-Based Injection

  • Prompted generations had variance-to-mean dispersion ratios of 0.6–1.0, compared with 1.15–1.29 for rule-based injection.The paper describes spontaneous human speech as over-dispersed, with V/M > 1.0.Source: §5.1 · pp. 6–7

5.2 Interruption and Backchanneling

  • τ-Voice produced 0.633 interruptions per agent turn but only 0.038 backchannels per turn, with a 1.02% opportunity conversion rate.Its 1,277 classifier consultations across 20 calls incurred continuous API billing while yielding little realized backchannel behavior.Source: §5.2 · pp. 7–8
  • At β = 0.3, interruptions took the first available mid-speech pause in 13.9% of instances; at β = 1.0, they did so in 56.1%.The patience threshold changes where interruptions occur relative to turn onset.Source: §5.2 · p. 8
  • At β = 1.0, qualifying mid-turn pause opportunities fell to 194, versus 230 at baseline and 246 at β = 0.7; the authors call this a weak effect.Executed interruptions terminate turns and therefore remove pauses that might otherwise follow.Source: §5.2 · p. 8

6 Conclusion

The paper audits gaps between configured and realized naturalness behaviors and proposes rule-based controls. Prompted and LLM-decided behaviors show operational shortfalls, while rule-based injection offers controllable rates and wider disfluency diversity without latency or extra model invocation costs.

6 Conclusion

  • Prompted disfluencies underdeliver, saturate rapidly, cluster at turn onset, and omit self-initiated same-turn repairs, while LLM-decided controls interrupt aggressively and nearly suppress backchannels.The LLM-decided controls also incur continuous API polling costs.Source: §6 · p. 8
  • Rule-based injection restores monotonic control, distributes disfluencies across utterances with natural over-dispersed turn-to-turn variance, and eliminates latency and extra model invocation costs.The paper notes that realized behavior remains tied to the target speaker’s speech cadence.Source: §6 · p. 8
  • LLM performance varies greatly across models, and the authors identify targeted spoken-dialogue training as a promising future direction for fine-grained naturalness control.The paper presents deterministic rule-based mechanisms as a current alternative.Source: §6 · p. 8

Limitations

The study’s limitations include untested interactions between interruption and backchanneling, thresholds calibrated to one agent, and a restricted evaluation scope. Its appendices also document the prompts and decision criteria used for disfluency and turn-taking.

Limitations

  • Interruption and backchannel parameters were tested separately; interruption is evaluated first and can truncate a turn before later backchannel opportunities arise.Calibration also used an agent with a median unbroken speech duration of 7.9 seconds, so thresholds may not transfer to substantially faster or slower agents.Source: Limitations · p. 9
  • Word-boundary insertion can split tightly bound syntactic units, and the rule-based design cannot produce complex self-initiated repairs without phrase-level substitution logic.Source: Limitations · p. 9
  • Turn handling uses 200 ms mid-speech-pause and 500 ms turn-end thresholds, both longer than the cited 110–130 ms human transition gaps.The authors attribute this real-time constraint to cascaded audio pipelines and production transport delays.Source: Limitations · p. 9
  • The evaluation covers only three models, a cascade architecture, specific prompts, and one fixed agent, so findings may not hold under substantially different conditions.The authors describe the rule-based approach as promising but incomplete and call for validation under more conditions.Source: Limitations · p. 9

A Verbatim Disfluency Prompt

  • τ-Voice’s disfluency prompt varies only its requested frequency, while asking for fillers, repetitions, false starts, and pauses to be naturally distributed.Source: A Verbatim Disfluency Prompt · p. 11
  • 1.22, 0.94, and 0.94 were the as-published-to-ablated rate ratios for gpt-4o-mini, gpt-5, and Gemini, respectively; removing the conflicting line did not sharply raise rates.The ablation removed only the appendix’s “MINIMAL VERBOSITY” line.Source: A Verbatim Disfluency Prompt · p. 11

B τ-Voice’s turn-taking decision prompts

  • τ-Voice uses separate LLM classifiers for interruption and backchanneling, with interruption taking priority; its prompts assess whether the user has heard enough and has a response ready.Backchannel guidance aims for roughly one continuer per four to six sentences of extended agent speech and allows silence when in doubt.Source: B τ-Voice’s turn-taking decision prompts · pp. 11–12

C Disfluency Full Results

Disfluency dispersion is summarized by comparing the variance of per-turn counts with their mean, which distinguishes more even placement from clustering.

D Dispersion Between Utterances for Disfluency Counts

E Statistical Tests

The statistical tests use paired scenario counts for interruption and backchanneling and per-turn observations for disfluency. The reported monotonicity of interruption and backchannel settings is descriptive, not tested between adjacent settings.

E Statistical Tests

  • Interruption and backchannel conditions use two-sided paired Wilcoxon signed-rank tests on per-scenario counts; intervals use 1,000 paired-bootstrap iterations.The test discards zero differences; pause events serve as a negative control, with none of the six conditions moving them.Source: E Statistical Tests · p. 12
  • Monotonicity across interruption and backchannel settings is observed rather than statistically tested, because comparisons are against baseline only.Source: E Statistical Tests · p. 12
  • Disfluency rate tracking uses Kendall’s τb across 1,196 paired per-turn observations per comparison; τ-Voice is excluded because its prompt is not a stronger setting of this study’s dial.Source: E Statistical Tests · p. 12
  • Table 3 reports disfluency rates per 1,000 words across conditions and models, including combined filler-plus-repetition rates, initial-filler share, and turns with any disfluency.Source: Table 3 · p. 13
  • Table 4 reports per-turn filler-plus-repetition variance-to-mean ratios; its 95% bootstrap interval distinguishes conditions from independent constant-rate placement when it excludes 1.0.Source: Table 4 · p. 13
  • 43% of positions were identical at all four injection settings, and ties made injection’s Kendall’s τb the lowest despite monotonicity by construction.The generator is seeded per utterance, so short turns can render identically at adjacent settings.Source: E Statistical Tests · p. 13

F Examples of Disfluency Generation

The examples contrast prompted disfluencies with rule-based injection in two reply positions. They also distinguish edits to a shared recorded reply from separately generated prompted replies.

F Examples of Disfluency Generation

  • An em dash signals a short pause, but its spoken realization depends on the speech synthesis model and is outside the study's scope.Source: F Examples of Disfluency Generation · p. 13
  • Table 5 reports paired comparisons against baseline across 20 scenarios per condition, while Table 6 tests configured strength against realized filler-plus-repetition rate using Kendall’s τb.Source: Table 5 · p. 14, Table 6 · p. 14
  • Injected rows edit one recorded reply, whereas prompted rows are separate generations that differ in content as well as disfluency.Source: F Examples of Disfluency Generation · p. 14
  • At nat_i1/8, prompted replies add fillers or repetitions, while injection places fillers and pauses into the recorded reply; at nat_i3/13, injection also adds repetitions and a restart.Source: F Examples of Disfluency Generation · p. 14

G Ethical Considerations

The work presents auditing practices and naturalness controls for evaluating real-time voice agents. It describes potential benefits for voice-system development alongside dual-use risks and mitigations.

G Ethical Considerations

  • The approach and evaluation practices target assessment of real-time voice agents over full-duplex telephony transport.Source: G Ethical Considerations · p. 15
  • The authors say reliable evaluation practices and controls can support safer, more resilient voice systems, while rule-based algorithms reduce LLM use and increase simulation efficiency.Source: G Ethical Considerations · p. 15
  • The controls could make spam or social-engineering robocalls more convincing and cheaper; the authors advocate responsible deployment and transparency, including disclosure of automated voice systems.Source: G Ethical Considerations · p. 15
Loading 2610.11015v1…