PaperlayerView exact revision on arXiv

Source-linked AI summary

SALMONN-duo: Adaptive Dual-System Coordination for Full-Duplex Voice Agents

Wenyi Yu, Siyin Wang, Terumi Chiba, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Lu Lu, Chao Zhang

arXiv:2609.34247v1cs.CLeess.AS

TL;DR

Full-duplex voice agents must combine low-latency conversation with reasoning and tool use whose costs vary by request. SALMONN-duo coordinates an always-on speech frontend with an asynchronous backend, training the frontend to delegate selectively. The paper reports improved QA performance–cost trade-offs and adaptive task completion and safety on τ-Voice.

  • Problem

    Voice agents must reconcile low-latency conversation with knowledge-intensive reasoning and environment tasks that require variable computation and tool use.

    Source: §1 · p. 1

  • Method

    SALMONN-duo pairs an always-on full-duplex speech frontend that learns when to answer or delegate with an asynchronous backend, trained using knowledge-boundary-aware SFT and cost-aware GRPO.

    Source: §1 · pp. 2–3

  • Results

    SALMONN-duo reports a favorable performance–cost trade-off on single- and multi-turn QA, adaptive task completion on τ-Voice, and improved task success and safety without substantially increasing delegation.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    The evaluations demonstrate that adaptive backend use can support knowledge-based answers and environment-grounded, policy-constrained multi-turn tasks while keeping the frontend responsive.

    Source: §1 · p. 3, §3.1.1 · p. 4, §6 · p. 10

    The delegation comparison assumes that system 2 is more capable in real-world settings.

    Source: §5.1 · p. 7

Abstract

from arXiv · show

Full-duplex speech large language models (LLMs) enable low-latency, natural voice interaction. However, real-world agents must also use tools and perform deliberative reasoning-operations whose variable latency and computational cost conflict with the stringent timing requirements of real-time conversation. To reconcile these demands, we propose SALMONN-duo, an adaptive dual-system voice agent inspired by dual-process theories of cognition. SALMONN-duo separates real-time interaction from deliberative computation by pairing an always-on, fast-thinking full-duplex speech LLM (system 1) with a powerful asynchronous slow-thinking LLM agent (system 2). Beyond handling real-time interaction, system 1 learns when to answer directly and when to delegate, remaining responsive during backend execution and seamlessly integrating returned information into the ongoing dialogue without exposing tool traces or losing conversational context. Evaluations on single-turn spoken question answering (QA) and multi-turn conversations demonstrate that adaptive delegation substantially improves accuracy on knowledge-intensive and multi-hop reasoning questions, while knowledge-boundary-aware training avoids unnecessary system 2 invocations. On a customized version of $\tau$-Voice, SALMONN-duo further demonstrates its ability to complete environment-grounded, policy-constrained tasks through multi-turn interactions in realistic business scenarios. Finally, cost-aware reinforcement learning further enhances the trade-off between task performance and backend usage across the QA and conversation tasks, while improving task success and response safety on $\tau$-Voice with an acceptable increase in the delegation rate.

1 INTRODUCTION

Full-duplex agents must reconcile low-latency conversation with variable-cost reasoning and tool use. SALMONN-duo pairs an always-on frontend with an asynchronous backend, training the frontend to delegate selectively; evaluations report a favorable performance–cost trade-off on QA and adaptive task completion on τ-Voice.

1 INTRODUCTION

  • Real-time responsiveness alone is insufficient when voice agents also need fresh or long-tail knowledge, multi-step reasoning, private environment state, or policy-constrained actions.Conversation demands consistently low latency, while task-solving computation varies with request difficulty.Source: §1 · p. 1
  • Existing full-duplex tool-call designs can create tension between real-time efficiency and task-solving capacity, while lightweight models may struggle with complex reasoning and tool use.Scaling the model for occasional complex requests also raises the computational cost of routine interactions.Source: §1 · pp. 1–2
  • SALMONN-duo pairs an always-on full-duplex speech LLM with an asynchronous backend, delegating complex tasks while the frontend stays engaged with the user.The backend may be a single LLM agent or a coordinator of specialist agents and tools.Source: §1 · p. 2
  • Knowledge-boundary-aware SFT trains the frontend to delegate knowledge questions when it needs a reference answer, while real-world task training covers information gathering and policy-based confirmation.Cost-aware GRPO further targets delegation utility, task success, and safety while limiting unnecessary backend use.Source: §1 · pp. 2–3
  • The authors describe SALMONN-duo as the first open-source work to explicitly consider the utility of backend invocations in dual-system coordination.They also identify multi-turn, task-oriented evaluation as a gap in prior open-source voice-agent evaluations.Source: §1 · pp. 2–3
  • Evaluations report a favorable performance–cost trade-off on single-turn and multi-turn knowledge-based QA, with adaptive tool use completing realistic τ-Voice tasks.Cost-aware GRPO further improves task success and safety without substantially increasing delegation.Source: §1 · p. 3

2 RELATED WORK

Prior work addresses full-duplex audio processing, tool use by voice agents, and cost-aware routing in text systems. SALMONN-duo brings asynchronous delegation and utility optimization into full-duplex voice interaction, where answers must be integrated as conversation unfolds.

2.1 FULL-DUPLEX SPEECH LARGE LANGUAGE MODELS

  • Full-duplex speech LLMs process user speech while generating assistant responses, using either specialized stream-compression architectures or interleaved time blocks.Recent work has also begun exploring reasoning capabilities in full-duplex models.Source: §2.1 · p. 3

2.2 TOOL USE AND DELEGATION IN VOICE AGENTS

  • Voice-agent tool-use approaches either add a dedicated tool-call branch or asynchronously invoke other models; SALMONN-duo trains a full-duplex frontend to invoke a backend agent when needed.The paper identifies invocation utility as an underexplored aspect of voice-agent coordination.Source: §2.2 · p. 3

2.3 UTILITY-AWARE ROUTING AND DELEGATION

  • Text-based routing work optimizes model selection or delegation using capability awareness, downstream outcomes, and cost, but full-duplex agents must also meet real-time constraints and coherently present answers during conversation.SALMONN-duo explores combining SFT and GRPO to optimize delegation utility for voice agents.Source: §2.3 · p. 3

3 METHODOLOGY

SALMONN-duo separates real-time interaction from asynchronous task execution: system 1 decides whether to delegate, and system 2 interprets the request and performs the task. Knowledge-boundary-aware SFT and cost-aware GRPO train the frontend to delegate when useful while remaining responsive in multi-turn interaction.

3.1 SYSTEM DESIGN

  • System 1 handles full-duplex interaction and alone decides whether to delegate; system 2 uses an invocation-time context snapshot to infer intent and carry out the task.The design pairs a real-time frontend with an asynchronous backend.Source: §3.1 · p. 4, Figure 2 · p. 4

3.1.1 SYSTEM 1: FULL DUPLEX REAL-TIME INTERACTION FRONTEND

  • System 1 builds on SALMONN-omni’s streaming speech encoder, LLM, and synthesizer, interleaving user and assistant streams into 80-ms time blocks.SALMONN-duo removes the echo stream and inserts system 2 output into the frontend by adding N tokens per time block, with N = 100.Source: §3.1.1 · p. 4

3.1.2 SYSTEM 2: ASYNCHRONOUS BACKEND AGENT

  • A suitable backend can be integrated plug-and-play if it interprets requests and returns reference answers or task-status updates in a form system 1 can use.For the text-based backend in this work, streaming ASR transcribes speech for conversation-history construction.Source: §3.1.2 · p. 4

3.1.3 DELEGATION MECHANISM

  • System 1 decides whether to delegate at each time block when it starts speaking, using either an additional head over intermediate LLM hidden states or a generated DELEGATE token.The frontend provides filler utterances while awaiting system 2, and subsequent response ordering can adapt to evolving user instructions.Source: §3.1.3 · p. 5

3.2 KNOWLEDGE BOUNDARY AWARE SUPERVISED FINE-TUNING

  • Knowledge-boundary-aware SFT labels delegation according to whether system 1’s generated answers are correct across N rollouts, adding contextually appropriate filler sentences when assistance is needed.The multi-turn daily-conversation training data is created in a similar way.Source: §3.2 · p. 5

3.3 COST-AWARE GROUP RELATIVE POLICY OPTIMIZATION

  • Cost-aware GRPO rewards task performance while penalizing unnecessary delegation, addressing exposure bias and evolving knowledge boundaries in long-horizon conversations.For task-oriented dialogue, training uses multi-turn episodes and adds a safety reward because binary task-success rewards are too sparse and failures can involve fabricated facts, misrepresented backend information, or policy violations.Source: §3.3 · p. 5

4 EXPERIMENTAL SETUPS

The experiments use a streaming speech frontend with separate LLM backends for knowledge QA and grounded τ-Voice tasks. Training combines large QA, conversation, and task-episode datasets; evaluation spans spoken QA, multi-turn dialogue, and task-oriented interaction.

4.1 MODEL SPECIFICATIONS

  • System 1 uses SPEAR, Llama-3.1-8B-Instruct, and an adapted CosyVoice2-0.5B streaming synthesizer, while backend models vary by task.The QA backend is gpt-4o-2024-11-20; τ-Voice uses gpt-5.2-2025-12-11 with domain-specific environment and tools.Source: §4.1 · p. 6

4.2 TRAINING SPECIFICATIONS

  • SFT uses 309k single-turn QA samples, 36k multi-turn conversations, and 32k τ-Voice episodes; cost-aware GRPO reuses SFT questions and uses 1,982 τ-Voice tasks.Multi-turn conversations contain 10–15 turns, and training runs on 64 A100-80G GPUs.Source: §4.2 · p. 6

4.3 EVALUATION SPECIFICATIONS

  • Evaluation covers spoken QA, 923 multi-turn dialogues of 3–5 turns, and τ-Voice task-oriented dialogue, with added questions to test response ordering during interruptions.Unless otherwise specified, the backend receives ground-truth dialogue history because ASR is not the focus.Source: §4.3 · p. 6

5 EXPERIMENTAL RESULTS

SALMONN-duo’s adaptive delegation performs strongly across spoken QA settings, while cost-aware training further improves routing and answer agreement. On τ-Voice, training with safety and cost rewards supports multi-turn task completion, and the model handles interruptions in the intended response order.

5.1 ANALYSIS OF THE DELEGATION MECHANISM

  • Among the compared delegation mechanisms, incorporating the delegation token into the response performs best when system 2 is unavailable and generally yields higher AUROC and answer-reference agreement.The authors adopt this mechanism in subsequent experiments.Source: §5.1 · p. 6, Table 1 · p. 7

5.2 SINGLE- AND MULTI-TURN KNOWLEDGE-BASED QUESTION ANSWERING

  • SALMONN-duo achieves strong spoken-QA performance while delegating less often on easier benchmarks than on more challenging ones.The comparison covers half-duplex and full-duplex models and voice agents; the authors report strong performance with system 2, even outperforming larger turn-based models on most datasets.Source: §5.2 · p. 7, Table 2 · p. 7
  • In multi-turn conversations, SALMONN-duo selectively invokes the backend and achieves higher delegation utility.Evaluation at turn N provides ground-truth context for the first N −1 turns.Source: §5.2 · pp. 7–8, Table 3 · p. 8
  • Cost-aware GRPO improves routing AUROC and final-response agreement with references in both single-turn and multi-turn QA, with gains across nearly all delegation budgets.The authors report these gains in Table 4 and Figure 3.Source: §5.2 · p. 8, Table 4 · p. 8, Figure 3 · p. 8

5.3 ENVIRONMENT-GROUNDED TASK-ORIENTED DIALOGUE

  • On τ-Voice, safety rewards improve task pass rate while reducing hallucinations and policy violations; adding a cost reward slightly lowers safety but preserves high pass rates.The cost reward also restores effective delegation to a level comparable to the SFT model; the authors consider increased tool use acceptable when it helps complete tasks.Source: §5.3 · p. 9, Table 5 · p. 9
  • SALMONN-duo handles interruption questions and original requests in the intended order, with interruption-question accuracy largely unaffected.The expected order depends on the user’s instructions and when system 2 returns sufficient information.Source: §5.3 · p. 9, Table 6 · p. 9

6 CONCLUSION

SALMONN-duo pairs an always-on full-duplex frontend with an asynchronous backend, training the frontend to delegate when it lacks sufficient knowledge or needs tools. The paper reports improved QA performance-cost trade-offs and successful environment-grounded task completion, with cost-aware reinforcement learning improving outcomes across its evaluations.

6 CONCLUSION

  • Knowledge-boundary-aware SFT and cost-aware GRPO train system 1 to invoke system 2 when it lacks sufficient knowledge or needs tools for environmental interaction.The frontend coordinates with an always-on system 1 and an asynchronous system 2 backend agent.Source: §6 · p. 10
  • On knowledge-based QA, SALMONN-duo outperforms recent half-duplex and full-duplex speech LLMs while achieving a better performance–cost trade-off than existing voice agents.Source: §6 · p. 10
  • On τ-Voice, SALMONN-duo addresses customer requests in realistic service scenarios, uses tools appropriately, handles barge-ins, and accomplishes users’ goals.Cost-aware reinforcement learning also improves task success and response safety, with an acceptable increase in delegation rate.Source: §6 · p. 10

A KNOWLEDGE BOUNDARY AWARE DATA GENERATION PIPELINE

The pipeline constructs single-turn and multi-turn QA training targets using model correctness and supervised repair, then simulates variable backend latency for training and evaluation.

A.1 SINGLE-TURN QA TARGETS

  • Single-turn QA targets are direct answers or DELEGATE-prefixed responses, chosen using the correctness of three sampled Llama-3.1-8B-Instruct answers per question.Delegated examples pair a contextually appropriate filler with a reference-based final answer.Source: §A.1 · p. 14

A.2 MULTI-TURN CONVERSATION SCENARIOS

  • Multi-turn scenarios begin from a topic and target 10–15 user-assistant rounds, with ordinary conversation, topic shifts, and simulated barge-ins represented before filtering.For barge-ins, the simulated user receives a prefix shorter than half of the preceding assistant utterance; the canonical history retains the full utterance.Source: §A.2 · p. 14
  • A user simulator generates utterances, an assistant simulator replies without an explicit topic field, and a supervisor judges the latest exchange for query status and answer correctness.A replacement is generated only when the turn is a query and the initial reply is incorrect.Source: §A.2 · pp. 14–15
  • For incorrect query replies, reference-guided repair replaces the assistant turn with a context-sensitive filler and an answer conditioned on the supervisor’s reference.The repaired reply is restored to the canonical history before the next round is generated.Source: §A.2 · p. 15

B SIMULATED BACKEND INVOCATION LATENCY

  • For knowledge QA, simulated retrieval latency is sampled relative to the initial response segment that does not rely on external knowledge.For lead segments of at least two seconds, sampling reserves at least one second between retrieval completion and the knowledge-dependent segment with 80% probability; shorter leads use uniform sampling over the full segment.Source: B SIMULATED BACKEND INVOCATION LATENCY · p. 15
  • For environment-grounded tasks, total delegation latency is the sum of independently sampled call latencies, each drawn uniformly from 1.5 to 2.5 seconds.The interval is based on the approximately two-second duration of one call to the experiment’s gpt-5.2-2025-12-11 backend.Source: B SIMULATED BACKEND INVOCATION LATENCY · p. 15

C.1.1 DATA SPECIFICATIONS

The authors specify the datasets and training examples used for knowledge-boundary-aware and τ-Voice training, alongside the training schedule and compute setup.

C.1.1 DATA SPECIFICATIONS

  • The pipeline produced 309k spoken question-answering examples—125k from TriviaQA, 94k from Natural Questions, and the remainder from HotpotQA—and 36k multi-turn dialogues drawn from the same three datasets.In single-turn QA, 113k examples (36.6%) require invoking system 2. In multi-turn dialogues, 61.2% of assistant turns respond to user knowledge-based questions, and 13.4% of those require invoking system 2.Source: §C.1.1 · p. 16
  • τ-Voice training combines 3.8k episodes generated from non-test tasks with 28.5k samples from the released SFT split, for approximately 32k examples.The backend simulator’s state and tool use are hidden from both user and assistant simulators during episode generation.Source: §C.1.1 · p. 16

C.1.2 OTHER DETAILS

  • Training warms up for 20k SFT steps, continues for 1.5k longer-audio steps, then applies cost-aware GRPO to all modules except the speech synthesizer.The GRPO stage uses a 2:1 GRPO-to-SFT loss weighting; pass, safety, and cost reward weights are 1, 1, and 0.5.Source: §C.1.2 · p. 16

C.2 EVALUATION DETAILS

The evaluation compares speech models and voice agents on spoken QA, multi-turn knowledge questions, and customized τ-Voice tasks. It also tests response ordering when users interrupt while system 2 is working.

C.2 EVALUATION DETAILS

  • Baselines include half-duplex and full-duplex speech LLMs, plus full-duplex voice agents MoshiRAG and VoiceChat.Half-duplex models begin processing after a user turn ends; full-duplex models can listen and speak simultaneously.Source: §C.2 · p. 16
  • Spoken QA is evaluated on Llama Questions, Web Questions, TriviaQA, HotpotQA, and HaluEval using transcribed model audio and a language-model judge.The OpenAudioBench datasets use official judge prompts; HotpotQA and HaluEval use the prompts from MoshiRAG.Source: §C.2 · pp. 16–17
  • The MultiTurn test set uses 3–5-turn dialogues and scores the final response to a knowledge-based request after earlier turns are provided as context.Source: §C.2 · p. 17
  • The evaluation also reports AUROC and Answer–Reference Agreement Rate, with AUROC’s no-delegation boundary varying by each model’s ability to answer without a reference.Agreement is assessed with the reference available by checking whether the final answer matches it in correctness.Source: §C.2 · p. 17
  • Customized τ-Voice evaluation disables interruption and backchannel checks, removes background noise and speaker variation, and uses only the test split.The training data uses the τ 2-Bench training split; evaluation retains gpt-5.2-2025-12-11 as user simulator and backend.Source: §C.2 · p. 17
  • The user-interrupt task evaluates whether the assistant conveys system 2 results amid evolving dialogue, with response order depending on user intent and result arrival.Table 7 describes expected response ordering according to whether system 2 returns its result before or after the interrupting question finishes.Source: §C.2 · p. 17, Table 7 · p. 17

D.1 EXTENDED ANALYSIS OF THE DELEGATION MECHANISM

Additional single- and multi-turn QA results are consistent with the main-text analysis, suggesting the delegation-token mechanism can achieve better performance.

D.1 EXTENDED ANALYSIS OF THE DELEGATION MECHANISM

  • Additional single- and multi-turn knowledge-based QA results are consistent with the main-text analysis, suggesting the delegation-token mechanism can achieve better performance.Source: §D.1 · p. 17

D.2 EXTENDED RESULTS OF COST-AWARE GRPO ON KNOWLEDGE-BASED QA

This section presents extended results for cost-aware GRPO on knowledge-based QA, including the effects of reference-answer latency and ASR errors, and describes evaluation prompts and criteria for correctness, grounding, and policy compliance. It also outlines a Codex-backed demo that handles background work and reports outcomes in spoken replies, with policy adherence assessed through explicit violations.

D.2 EXTENDED RESULTS OF COST-AWARE GRPO ON KNOWLEDGE-BASED QA

  • Cost-aware GRPO further improves the model’s delegation utility on knowledge-based question-answering tasks, with additional results reported for the Llama Questions and Web Questions datasets.Source: §D.2 · p. 17

D.3 SENSITIVITY TO REFERENCE RETRIEVAL DELAY

  • VoiceChat remains relatively stable as reference-retrieval latency varies, while MoshiRAG exhibits substantial performance fluctuations.System 2 reference answers are inserted directly into time blocks as frontend context; VoiceChat rarely requests references, limiting latency effects.Source: §D.3 · p. 18

D.4 SENSITIVITY TO ASR CORRECTNESS

  • ASR transcription accuracy affects overall performance, but backend reference correctness is more critical and can persist despite errors when core meaning is recovered.The authors note ASR performance matters for real-world deployment and leave systematic investigation of its impact to future work.Source: §D.4 · pp. 18–19

E.1.3 PROMPTS FOR SUPERVISOR SIMULATOR

  • The supervisor simulator judges whether the latest user message seeks an answer and whether the assistant responds correctly, using earlier turns as context.For queries, it also supplies a concise, factually sound reference answer; non-queries receive null values for correctness and reference.Source: §E.1.3 · pp. 23–24

E.2 MULTITURN EVALUATION PROMPT

  • Multiturn QA scoring accepts equivalent core meanings and valid aliases, interpreting commas in context rather than requiring exact string matches.Numbers may be written as numerals or words, and punctuation or simple spelling errors may be ignored.Source: §E.2 · pp. 24–25

E.3 HALLUCINATION EVALUATION PROMPT

  • The hallucination auditor flags material unsupported facts or misrepresentations of tool results, using only evidence available by the time of each assistant message.A single demonstrated material failure makes the episode score 0; genuine ambiguity is resolved in favor of score 1.Source: §E.3 · pp. 26–27

E.4 POLICY ADHERENCE EVALUATION PROMPT

  • The policy auditor checks only observable conduct against explicit policy requirements, excluding factual-grounding errors unless the policy explicitly addresses them.It treats successful backend results as authoritative for policy prerequisites and does not let later corrections erase earlier demonstrated violations.Source: §E.4 · pp. 27–28
  • The rubric assigns 0 only for a demonstrated violation of an explicit policy requirement, and 1 otherwise, resolving evidentiary ambiguity in favor of 1.Factual errors, unfinished tasks, honest backend failures, and justified refusals count as 1 unless an explicit policy rule prohibits them.Source: §E.4 · pp. 28–29

F REAL-WORLD CASE STUDY

  • A per-conversation Codex coordinator manages background task agents, while a local orchestrator validates their actions and system 1 speaks from the actual outcomes.The coordinator can generate reference information directly or launch, update, query, or cancel tasks; system 1 uses the resulting reference to formulate its reply.Source: F REAL-WORLD CASE STUDY · p. 30

F.1 REQUESTS REQUIRING UP-TO-DATE KNOWLEDGE

  • For an up-to-date-knowledge request, SALMONN-duo acknowledged the user while a task agent checked FIFA sources and created a document, then reported completion.The user confirmed that the file was present.Source: F REAL-WORLD CASE STUDY · p. 30, Figure 5 · p. 31

F.2 CODE REVIEW AND MODIFICATION

  • For a code-debugging request, a task agent corrected a comparison in sort.py and passed the sample input plus six additional sorting cases before system 1 summarized the result.The example illustrates an interim response followed by a spoken reply based on the reported task outcome.Source: F REAL-WORLD CASE STUDY · p. 30, Figure 6 · p. 32

Related papers

Loading 2609.34247v1…