PaperlayerView exact revision on arXiv

Source-linked AI summary

DuplexSpeechBench-IFEval: Evaluating Implicit Instruction Following in Full-Duplex Voice Agents

Puneet Mathur, Dinesh Manocha

arXiv:2609.03423v1cs.AI

TL;DR

Full-duplex voice agents need to infer role-appropriate conversational behavior, yet existing evaluations largely test explicit turn-management instructions. The paper introduces DSB-IFEval to measure implicit instruction following across conditioning protocols and systems, finding that persona inference, real-time execution, and conflict resolution remain distinct capabilities.

  • Problem

    Full-duplex agents must infer which conversational behavior a role requires and execute it at the right moment, beyond following explicit turn-management instructions.

  • Method

    DSB-IFEval evaluates 1,038 test cases across eight roles and five conditioning protocols, using IAS for floor management and PAS for persona-consistent content.

  • Results

    Across six systems, persona-consistent content and real-time floor control are distinct, architecture-dependent capabilities, and reliable directive following does not guarantee correct behavior under conflict.

  • Takeaways & Limitations

    Effective persona-conditioned agents must understand a role, translate it into appropriate behavior, execute it at the right time, and resolve competing instructions.

  • Takeaways & Limitations

    The benchmark is English-only, limited to two-turn interactions, and uses controlled scripted TTS speech that does not capture the full variability of live human dialogue.

Abstract

from arXiv · show

Full-duplex voice agents must continuously decide when to listen, backchannel, interrupt, handle speech overlaps, take the floor, and yield. Existing benchmarks largely test these behaviors through explicit turn-management instructions, while deployed agents are often configured through roles or personas from which the appropriate conversational behavior must be inferred. We introduce DuplexSpeechBench-IFEval (DSB-IFEval) for evaluating implicit instruction-following in real-time spoken interaction. (DSB-IFEval) comprises 1,038 test cases spanning eight diverse assistant roles and evaluates five conditioning protocols for instruction-following: default behavior, explicit behavioral instructions, persona-implied behavior, combined persona--rule conditioning, and instruction conflict. We measure real-time floor management using a deterministic Instruction Adherence Score (IAS) and persona-consistent content using LLM-judged Persona Adherence Score (PAS). Across six real-time speech systems, we find architecture-dependent trade-offs. Full duplex models like F-Actor and PersonaPlex are more sensitive to whether conversational behavior is stated explicitly or must be inferred from a persona, with adherence dropping by 9.7% and 4.5%, respectively, under persona-only conditioning. In contrast, GPT-Realtime, MiniCPM-o, and Fun-Audio-Chat strongly adhere to persona-consistent content, but their floor behavior does not adapt across explicit and persona-only instructions and remains constrained on several proactive actions. We further find that even if systems reliably follow conflicting directives to their prescribed persona, they still struggle to override them under safety conflict. These results show that inferring the behavior implied by a role, executing it at the appropriate conversational moment, and resolving competing instructions remain distinct challenges for full-duplex voice agents.

1 Introduction

Full-duplex agents must infer role-appropriate conversational behavior and execute it under real-time constraints, but existing evaluation often focuses on explicit instructions. DSB-IFEval jointly evaluates implicit and explicit behavior following, persona-consistent content, and instruction conflicts across real-time speech systems.

  • Motivation: Full-duplex agents must coordinate listening, backchannels, interruptions, overlap, floor-taking, and yielding according to their roles.A tutor, counselor, interpreter, and dispatcher may require different conversational actions and timing.
  • Benchmark: DSB-IFEval evaluates 1,038 test cases across eight assistant roles and five conditioning protocols for implicit and explicit instruction following.The protocols cover default behavior, explicit instructions, persona-implied behavior, combined persona–rule conditioning, and instruction conflict.
  • Findings: Persona-consistent content and real-time floor control emerge as distinct capabilities, so strong persona adherence does not guarantee persona-conditioned behavior.Systems may track a persona in responses while remaining constrained on proactive actions such as backchanneling and interruption.
  • Evaluation: DSB-IFEval separates real-time floor management, measured by deterministic IAS, from persona-consistent content, measured by LLM-judged PAS.This separation distinguishes role-appropriate language from execution of the corresponding conversational-floor behavior.
  • Findings: Across six systems, architecture shapes how persona conditioning affects floor control, proactive behavior, and content adherence.The evaluation also examines instruction conflicts, including benign conflicts and safety conflicts with different precedence rules.

2 Related Work

Prior benchmarks assess spoken instruction following and full-duplex interaction, but DSB-IFEval targets the additional challenge of inferring floor behavior from roles or personas and resolving competing instructions.

  • Full-duplex evaluation: Full-duplex benchmarks evaluate turn-taking, pauses, backchanneling, overlap, interruption handling, latency, robustness, multi-turn quality, instruction following, safety, and tool use.These benchmarks cover complementary aspects of real-time spoken interaction.
  • Speech instruction following: Speech and voice-agent benchmarks assess task, semantic, expressive, conversational, paralinguistic, and system-prompt instruction compliance.Examples include VoiceBench, SpeechInstructBench, S2S-Arena, VCB Bench, and CAVA.
  • DSB-IFEval: Unlike primarily explicit-instruction evaluations, DSB-IFEval tests whether floor-management behavior can be inferred from a persona.It also evaluates restated implied rules and conflicts between explicit and role-implied instructions.

3 DuplexSpeechBench–IFEval

DSB-IFEval evaluates full-duplex behavior under explicit, persona-implied, combined, and conflicting conditioning. Its controlled, role-contrastive design holds user-side interactions fixed while varying behavioral specification.

  • Benchmark Design & Taxonomy: DSB-IFEval tests whether agents execute stated policies, infer behavior from personas, and resolve conflicts between persona-implied and explicit directives.The benchmark covers listening, backchanneling, interruption, floor-taking, overlap, and yielding under varying behavioral specification.
  • Benchmark Design & Taxonomy: Five conditioning levels compare default behavior, explicit instruction, persona-only inference, persona-plus-entailed instruction, and conflicting instructions.L2 requires both inferring the appropriate behavior from the role and executing it during spoken interaction.
  • Dataset Generation: The benchmark reuses each generated spoken interaction across conditioning levels, isolating prompt and persona effects while holding user-side content fixed.The pipeline inserts structured timing markers and assembles independently synthesized speech with controlled pauses and gaps.
  • Dataset Generation: Eight behaviorally contrastive assistant roles vary expected actions such as interruption, backchanneling, silence tolerance, readback, and responses to overlap.The same conversational event can require preservation, intervention, or continued speech depending on the assigned role.
  • Dataset Generation: The generation pipeline expands 240 controlled conversations across five protocols into 1,038 evaluation cases, including a matched-audio subset.The conversations contain explicit timing events, controlled pauses, and overlap triggers.
  • Dataset Generation: Each role uses six probes: four shared structures for cross-role comparison and two role-specific probes for unique behavior traits.Probes create decision points involving silence, backchanneling, interruption, continued speech, yielding, and overlap.

4 Evaluation

The evaluation separates deterministic real-time floor management from persona-consistent content and quantifies how conditioning changes instruction adherence. It also evaluates outcomes when directives conflict with persona-implied or safety-relevant behavior.

  • Evaluation Metrics: Instruction adherence measures whether the model performs the expected full-duplex action at the correct time, while persona adherence measures role-consistent spoken content.The evaluation uses IAS for behavior and PAS for persona consistency.
  • Instruction Adherence: IAS deterministically scores nine target behaviors using known probe timestamps, speech activity, overlap, and floor-transfer timing.Probe-specific verifiers produce binary pass/fail outcomes, averaged across cases at each conditioning level.
  • Behavioral Effects: The ENTAILMENT GAP measures the cost of inferring behavior from a persona rather than receiving it explicitly.Positive values indicate better execution when the behavior is explicitly stated.
  • Behavioral Effects: Redundancy Gain measures whether restating persona-implied behavior improves execution, while Role Tax measures the effect of adding persona conditioning to explicit instructions.Negative Role Tax values indicate that persona conditioning reduces execution accuracy.
  • Persona Adherence: PAS uses an LLM judge to score persona-appropriate content and conversational register from 0-100.The judge receives the interleaved transcript together with the persona description.
  • Conflict Resolution: Conflicting conditions are categorized as directive-wins, persona-wins, balanced, or incoherent, with safety conflicts additionally reporting safety-over-directive behavior.The safety-relevant outcome is defined as the fraction of cases where persona-implied safety behavior takes precedence.

5 Experimental Setup

The experiment compares six real-time speech systems under a shared streaming and orchestration protocol. Model-specific adapters accommodate audio and streaming differences while preserving common user audio and event timing.

  • Systems: Six systems are evaluated: GPT-Realtime, MiniCPM-o-4.5, Fun-Audio-Chat, PersonaPlex, F-Actor, and Moshi.The systems span real-time-capable and synchronous full-duplex architectures.
  • Systems: Moshi serves as a persona-blind negative control because its floor behavior cannot depend on persona or explicit textual instruction.Its ENTAILMENT GAP provides a reference noise floor for the paired L1–L2 comparison.
  • Evaluation Protocol: All systems use the same pre-generated user audio and event timing across systems and conditioning levels.Model-specific adapters handle only audio encoding, sample rate, and streaming requirements.
  • Evaluation Protocol: Open-weight models run on H100 80 GB GPUs, while GPT-Realtime is evaluated through its streaming API.Multiple model instances run in parallel where supported.

6 Results

Across conditioning levels, systems trade off persona-consistent content against real-time floor control, with architecture shaping both adaptation and conflict handling. Persona-only inference can reduce floor adherence, while explicit conflict resolution does not ensure timely safety-preserving behavior.

  • Conditioning levels: F-Actor’s IAS falls from 35.5% under explicit instruction to 25.8% under persona-only conditioning, while PersonaPlex falls from 11.0% to 6.5%.These L1–L2 differences quantify the cost of inferring behavior from a persona.
  • Conditioning levels: GPT-Realtime, MiniCPM-o, and Fun-Audio-Chat show stronger persona-conditioned content with comparatively stable IAS across L1–L3.PAS rises substantially under persona-only conditioning for these systems, while floor behavior changes comparatively little.
  • Derived metrics: ENTAILMENT GAP measures the cost of persona inference, Redundancy Gain measures restatement benefits, and Role Tax measures persona effects on explicit instructions.F-Actor and PersonaPlex have the largest entailment gaps, while persona conditioning does not improve instruction following in any model.
  • Instruction conflict: Safety conflict reduces IAS to 0.0% for four of six systems; MiniCPM-o and Fun-Audio-Chat retain 30.0% adherence.The result indicates difficulty executing safety-preserving behavior at the required conversational moment.
  • Instruction conflict: Under safety conflict, MiniCPM-o selects the safety-preserving resolution in 60.0% of cases, compared with 43.3% for Fun-Audio-Chat and 33.3% for GPT-Realtime.Recognizing the correct hierarchy does not guarantee executing the corresponding full-duplex behavior at the required time.
  • Action and role heterogeneity: Action-level profiles reveal near-binary proactive behavior in MiniCPM-o and Fun-Audio-Chat, whereas F-Actor shows more graded floor control.F-Actor scores 33.3% on Interrupt, 72.5% on Take Turn, and 50.8% on Yield.

7 Discussion

The results indicate that implicit instruction following requires both inferring behavioral consequences from a role and executing them at the correct moment. Content adherence and full-duplex floor control therefore need separate evaluation, as does safety-aware instruction hierarchy.

  • Implicit instruction following: High persona adherence does not necessarily imply persona-conditioned floor behavior because default actions may already match the expected behavior.The ENTAILMENT GAP separates strong absolute adherence from sensitivity to whether behavior is explicitly stated or inferred.
  • Architectural effects: Turn-based systems show near-binary proactive actions, whereas full-duplex systems provide more graded control during ongoing speech.The discussion motivates reporting action-level results rather than relying only on aggregate scores.
  • Instruction hierarchy: Reliable directive following in benign conflicts does not guarantee overriding directives for role-implied safety behavior.Safety-aware instruction hierarchy is presented as a distinct capability for deployed voice agents.

8 Conclusion

DSB-IFEval evaluates implicit instruction following by varying prompts from explicit instructions to persona-only descriptions and conflicting directives. Across six systems, persona-consistent content and real-time floor control are distinct, architecture-dependent capabilities, and instruction following does not guarantee correct conflict resolution.

  • Conclusion: DSB-IFEval makes implicit instruction following measurable through explicit, persona-only, combined, and conflicting prompt conditions.The benchmark evaluates how systems translate roles into behavior during real-time interaction.
  • Conclusion: Strong persona adherence does not imply successful persona inference, and reliable directive following does not guarantee correct behavior when instructions conflict.Effective persona-conditioned agents must understand a role, translate it into timely behavior, and resolve competing instructions.

9 Limitations

The benchmark is intentionally narrow in language, interaction length, and speech source, limiting its coverage of natural dialogue and longer-term adaptation.

  • The benchmark is English-only and limited to two-turn interactions.It does not measure long-horizon persona drift, instruction revision, repeated-interaction adaptation, or multilingual and cross-cultural turn-taking norms.
  • Scripted TTS with controlled pauses provides exact ground truth and reproducibility but omits live human dialogue variability.The limitation concerns prosodic, acoustic, and behavioral variability not captured by the controlled speech input.

10 Ethical Considerations

The benchmark uses role-conditioned policies to test conversational behavior across contrasting settings, including safety-sensitive scenarios, but its ethical scope is limited to conversational policy evaluation.

  • Ethical scope: Safety-sensitive roles are included only to evaluate conversational policy behavior, not clinical or emergency decision-making.The paper explicitly limits claims about medical triage and emergency dispatch.
  • Role-conditioned expectations: The benchmark probes role-specific floor-management behavior through actions such as listening, taking the turn, backchanneling, yielding, interrupting, and readback.Its deterministic action vocabulary assigns each case an expected floor-management action and a temporal event.
  • Role-conditioned expectations: The eight roles are designed to induce contrasting expectations for interruption, backchanneling, silence, readback, and overlap handling.The benchmark spans roles such as meditation instructor, tutor, drive-thru order taker, triage nurse, bereavement companion, and emergency dispatcher.

E.7 Threshold Sensitivity Analysis

The analysis tests whether ENTAILMENT GAP rankings remain stable across verifier thresholds and finds that the ranking's top is substantially more stable than its fine ordering.

  • Threshold sweep: 180 verifier-threshold configurations are evaluated by sweeping take-turn, yield/continue, interrupt, and backchannel-duration thresholds.The sweep recomputes verdicts from stored measurements without rerunning VAD.
  • Ranking stability: Mean Kendall’s τ is 0.796 across threshold configurations, with a median of 0.867 and a range from 0.467 to 1.0.These values compare each threshold-based ENTAILMENT GAP ranking with the default-threshold ranking.
  • Ranking stability: F-Actor remains the top-ranked model in all 180 configurations, while the full six-way ordering is preserved in only 20%.The top-2 pair is preserved in 73%, and models with near-zero gaps are more likely to reshuffle.
  • Metric separation: IAS and PAS measure different properties: deterministic floor management versus role-consistent language and content.The PAS/conflict judge receives role and transcript information while timing and floor labels are withheld.

G Experimental Configurations

The experiments evaluate fresh model sessions using configured decoding controls, structured judge inputs, deterministic scoring, and targeted diagnostics for floor control, persona adherence, and safety conflicts.

  • Execution setup: Each model case runs in fresh state, with new sessions or reset streaming and message state across conditioning variants.This prevents cross-case history, cache, or turn-detector state from carrying between evaluations.
  • Diagnostics: Aggregate IAS is supplemented by action-level floor-control profiles, role-level ENTAILMENT GAP results, and matched-audio comparisons.These diagnostics expose behavior that aggregate scores can hide and reuse the same user waveform across selected role conditions.
  • Judge configuration: The evaluation records model transcripts and structured judge fields for persona adherence and conflict classification.The judge evaluates register and content separately from timing and returns PAS plus conflict labels where applicable.
  • Safety-conflict evaluation: The L4b safety-conflict subset targets cases where a literal directive conflicts with role-critical behavior.Six role–probe cells are included, with five directly negating safety-critical readback or interruption behavior.

I Reproducibility and Release Checklist

The benchmark release provides artifacts, evaluation settings, diagnostic tables, and qualitative examples needed to reconstruct conditions and inspect reported metrics.

  • Release artifacts: The release includes specifications, generated scripts, event manifests, scorers, timing measurements, judge outputs, analysis scripts, and runtime adapters for reproducibility.These artifacts support reconstruction of benchmark conditions and regeneration of automatic metrics.
  • Evaluation settings: Tables document evaluated-system configurations, including reset behavior for local systems and fresh hosted sessions for GPT-Realtime.
  • Evaluation settings: Inference settings are reported through the decoding controls explicitly configured by each evaluation adapter.
  • Diagnostics: IAS results are organized by expected action, with proactive-floor axes marked separately and extreme VAD-gated or turn-based values attributed to architecture.
  • Diagnostics: Role-level diagnostics report ENTAILMENT GAP in percentage points, while noting that individual cells should not be interpreted as persona inference in isolation.Positive values indicate higher IAS under explicit instruction than persona-only conditioning.
  • Diagnostics and examples: Additional tables provide matched-audio role diagnostics, verified qualitative examples, and a summary of release artifacts for benchmark reproduction.The matched-audio diagnostic holds the user waveform byte-identical while changing the assistant persona and expected behavior.
Loading 2609.03423v1…