PaperlayerView exact revision on arXiv

Source-linked AI summary

Qwen-Audio-3.1-Realtime: Towards Reliable Agentic Voice Interaction

Lujia Bao, Qian Chen, Luyao Cheng, Chong Deng, Yuxiang Kong, Xiangang Li, Xu Li, Jiaqing Liu, Chao-Hong Tan, Haoyu Wang, Wen Wang, Xilou Wang, Haoxiang Xu, Junhao Xu, Liang Yi, Binbin Zhang, Qinglin Zhang, Qiquan Zhang

arXiv:2609.25176v2eess.AScs.AIcs.CLcs.SD

TL;DR

Real-time voice assistants must handle evolving requests, actions, and conversational rules. Qwen-Audio-3.1-Realtime combines foundation training, executable action learning, and conversational policy alignment. Under the reported settings, it improves on 3.0 across several capability areas, including half-duplex task success.

  • Problem

    Real-time assistants must reason over evolving requests, execute actions, and follow conversational rules, including tracking goals and execution as circumstances change.

    Source: §1 · p. 2

  • Method

    The report connects Think, Act, and Speak and Coordinate through foundation post-training, self-evolving executable environments with multi-granularity feedback, and a policy for whether, when, and how to speak or act.

    Source: §1 · p. 2, §9 · p. 20

  • Results

    82.0% versus 78.4% overall task success on the half-duplex S2T τ-Voice adaptation; 3.1 also improves across several reported capability areas under the stated settings.

    Source: §6.3 · p. 12, §9 · p. 20

  • Takeaways & Limitations

    Further evaluation should test these capabilities together in long-horizon spoken tasks using real recordings and end-to-end latency measurements.

    Source: §9 · p. 20

    Most comparisons are descriptive; suite-wide confidence intervals, repeated-seed tests, and contamination analyses are unavailable.

    Source: §8 · p. 20

Abstract

from arXiv · show

Real-time voice assistants must reason over evolving requests, execute actions, and follow conversational rules. Qwen-Audio-3.1-Realtime brings these requirements together through Think, Act, and Speak and Coordinate. Think combines Core-Cocktail supervised fine-tuning with Multimodality and Multi-Teacher On-Policy Distillation (M$^{2}$-OPD) to transfer language capabilities and develop native audio skills. Act uses self-evolving executable environments and multi-granularity rollouts for Group Relative Policy Optimization (GRPO), teaching the model to use tools, interpret feedback, and complete tasks. Speak and Coordinate aligns whether, when, and how the assistant speaks or acts. We evaluate audio reasoning, multilingual understanding, tool use, conversational behavior, full-duplex interaction, and safety. Compared with Qwen-Audio-3.0-Realtime, 3.1 raises overall task success from 78.4% to 82.0% on our half-duplex speech-to-text adaptation of $τ$-Voice. On speech-to-speech Full-Duplex-Bench v1.5, the response rate to background speech falls from 73.0% to 13.0%. We also present a separate Voice Harness prototype, using Qwen-Audio-3.0-Realtime as its foreground, that extends spoken interaction to persistent tasks through foreground--background coordination and memory.

1 Introduction: Realtime Beyond Latency

Realtime voice interaction requires more than low latency: assistants must track changing requests, execute tools, interpret feedback, and follow conversational rules. The report organizes these needs into Think, Act, and Speak and Coordinate, with action learning at its methodological center.

1 Introduction: Realtime Beyond Latency

  • Assistants must retain constraints as requests evolve, choose and execute tools, interpret feedback, and follow user instructions and interaction rules.Low response latency is one requirement within this broader problem.Source: §1 · p. 2
  • Think supports audio reasoning and multi-turn instruction following; Act turns spoken intent into executable decisions under task and policy constraints.Speak and Coordinate governs whether, when, and how the assistant speaks or acts, including concurrent progress updates while a tool runs.Source: §1 · p. 2
  • Executable domains, construction checks, rollout-driven evolution, and dialogue-, milestone-, and turn-level feedback train action beyond function-call syntax.The stated targets include state verification, long-horizon completion, appropriate refusal, and progress communication.Source: §1 · p. 2

2 Qwen-Audio-3.1-Realtime Overview

The overview separates training dependencies from runtime interaction and describes concurrent action and speech. The architecture uses distinct paths for timing control and response content, and the release comparison is between the 3.0 and 3.1 systems.

2 Qwen-Audio-3.1-Realtime Overview

  • Foundation training supports agentic training and conversational policy alignment, while Think branches to concurrent Act and Speak at runtime.Figure 2 distinguishes foundation-support arrows from interaction-constraint arrows.Source: Figure 2 · p. 2, §2 · p. 3

2.1 Architecture and interfaces

  • Both runtime paths use an Audio Encoder + LLM: full-duplex decisions control speech timing, while the Speech-to-Text path generates response content for context-aware streaming voice rendering.The renderer uses conversation and acoustic context, guided by interaction decisions.Source: §2.1 · p. 3, Figure 3 · p. 3

2.2 Release scope

  • The release-scope comparison evaluates the released Qwen-Audio-3.0-Realtime and Qwen-Audio-3.1-Realtime systems within the modality, benchmark, and judge settings stated in Section 6 and the table captions.Source: §2.2 · p. 3

3 Foundation Training for Spoken Intelligence

Foundation training combines text-grounded and audio-conditioned supervision through a staged post-training recipe. It establishes an initial audio policy, applies two on-policy distillation stages, and consolidates domain expertise into one main model.

3 Foundation Training for Spoken Intelligence

  • The four-stage recipe uses Core-Cocktail, Multimodality OPD, domain GRPO specialists, and Multi-Teacher OPD to consolidate supervision into the main model.M2-OPD names the two OPD stages together, and only the consolidated main model is needed at inference.Source: §3 · pp. 3–4

3.1 Core-Cocktail supervised fine-tuning

  • Core-Cocktail is the cold-start stage: it reconciles shared language parameters before controlled SFT establishes an audio policy using joint audio and text instruction data.The broader recipe uses million-hour-scale paired audio-text data spanning audio understanding, spoken instruction following, interaction, tool-oriented requests, and other domains.Source: §3.1 · p. 4

3.2 Multimodality OPD with an Audio Reference

  • Multimodality OPD scores student-generated trajectories with a Text Teacher and frozen Audio Reference, adding audio-conditioned supervision for emotion, pragmatic intent, or acoustic context.The audio-conditioned signal targets information that may go beyond the transcript.Source: §3.2 · p. 5

3.3 Multi-Teacher OPD

  • Domain data and GRPO produce audio specialists, whose token-level distributions supervise the main AudioLM on its own audio-conditioned trajectories.Relevant experts receive the same audio context and student prefix; this consolidation stage differs from the separate executable-task GRPO used for action behavior.Source: §3.3 · p. 5

4 Learning to Act in Executable Environments

The section describes executable environments for learning reliable action under spoken task and policy constraints. Figure 5 connects domain and task construction, rollout scoring, environment evolution, and multi-granularity GRPO feedback.

4.1 Executable domains and tasks

  • Each executable domain combines tools, a stateful database, and policies, while each task defines an episode, its initial state, user script, expected outcome, behavioral assertions, and reference tool-call chain.Tasks distinguish compliant writes, justified refusals, and unsupported requests; informational tasks can be scored through behavioral assertions without a state change.Source: §4.1 · pp. 6–7

4.2 Running and scoring a rollout

  • Rollouts pair the model with a scripted user simulator, then score terminal state, permitted writes, and behavioral requirements against the episode’s tool trace and private state.A correct state does not excuse an interaction-rule violation, and a fluent reply cannot compensate for a failed state check or unauthorized write.Source: §4.2 · p. 7

4.3 Evolving domains from rollout results

  • Aggregated valid rollout outcomes label tasks as solved, failed, or near the capability boundary, guiding repairs, harder variants, and retention of frontier cases.Broken environments are repaired, while valid tasks exposing model weaknesses remain training data; feedback can also improve task-generation templates and checks.Source: Figure 5 · p. 6, §4.3 · p. 8

4.4 Three levels of reward signals for GRPO

  • GRPO uses dialogue-, milestone-, and turn-level feedback to compare complete task outcomes, continuations from saved intermediate states, and local decisions.Milestone continuations focus comparison on subsequent decisions; turn-level checks target tool choice, parameter grounding, unauthorized writes, and reply quality.Source: §4.4 · p. 8

4.5 Speech-conditioned tool use and retrieval

  • Speech-conditioned tool training uses synthesized user audio so spoken numbers, similar-sounding names, and negation affect tool selection and parameter grounding.Search training separates whether to search, what to query, and how to answer from evidence; its query reward penalizes submitting more queries than the judge-estimated reasonable count.Source: §4.5 · p. 9

4.6 Reporting progress before tool calls

  • Before tool calls, progress acknowledgments may be required, disallowed, or optional; evaluation checks their placement and properties while penalizing premature promises and repeated filler.The action layer determines what to do, while interaction policy governs how progress is communicated.Source: §4.6 · p. 9

5 Conversational Policy Alignment

Conversational policy alignment governs whether, when, and how the assistant speaks or acts. The section describes decision constraints for tool use and turn timing, alongside constraints and evaluations for spoken expression.

5.1 Whether to speak or act: boundaries and permissions

  • The assistant must choose among answering, using a tool, clarifying or confirming, refusing, and staying silent, while distinguishing learned policy following from permissions enforced by tools and runtime.A spoken promise is neither proof of execution nor an authorization token.Source: §5.1 · p. 9, §5 · p. 9

5.2 When to speak: waiting, yielding, and resuming

  • Turn timing depends on streaming audio, interaction history, and system state, with explicit current-session agreements taking priority over scene-level defaults while tool permissions and safety rules remain binding.Stopping playback and deciding whether to answer are distinct decisions; interruptions, backchannels, and speech addressed to others call for different responses.Source: §5.2 · p. 10

5.3 How to speak: expression, persona, and empathy

  • Spoken expression is shaped by style, persona, emotional appropriateness, and naturalness, while evaluation covers multi-scene dialogue, empathy, and persona under specified evaluation conditions.The empathy evaluations use a common high-empathy persona, and reported persona and empathy results use speech-to-text evaluation; larger human red-team studies and long-horizon executable spoken-task evaluation remain future work.Source: §5.3 · p. 10

6 Evaluation: Intelligence, Action, Interaction and Realtime

The evaluation compares Qwen-Audio-3.1-Realtime with 3.0 across intelligence, tool use, spoken interaction, duplex behavior, and safety, with results interpreted under each benchmark’s stated modality and protocol. Results improve across many tasks, while retrieval-triggering F1 slips, some conversational scores are mixed, and the 50-session red-team study remains limited in scope.

6.1 Protocol and evidence scope

  • Reported comparisons are descriptive and should be read within each table’s modality, data, judge, and sample-count setting.The τ-Voice results use a half-duplex S2T adaptation and are not directly comparable to its official full-duplex S2S protocol; release-level comparisons do not isolate individual training components.Source: §6.1 · pp. 10–11

6.2 Intelligence: general, audio, and long-context capabilities

  • Qwen-Audio-3.1-Realtime improves across audio reasoning, multilingual understanding, speech recognition, and long-context tasks, while VoiceChat-L remains close to 3.0.AMC rises from 47.12 to 52.21; multilingual BBA macro-average reaches 88.1, FLEURS macro-average WER falls from 9.01 to 3.98, and Long-AMC improves by 8.04 percentage points.Source: §6.2 · pp. 11–12, Table 2 · p. 11, Table 3 · p. 11

6.3 Action: tool use and retrieval

  • 82.0% overall task success, up from 78.4%, on the half-duplex S2T τ-Voice adaptation.The reported aggregate is not an unweighted average of the displayed domain scores and is not directly comparable to official full-duplex τ-Voice results.Source: §6.3 · p. 12, Table 6 · p. 13
  • On WebSearch1K, mean proposed queries fall from 4.37 to 1.05, while retrieval-triggering F1 decreases from 60.87% to 58.61%.The evaluation tests whether to search and which queries to emit, not retrieval execution, grounded-answer quality, latency, or measured search cost.Source: §6.3 · pp. 12–13, Table 7 · p. 13

6.4 Interaction: spoken behavior, persona, empathy, and duplex

  • On VoiceChat, 3.1 records the highest displayed overall Spoken and Rubric scores, while scene-level results include both gains and declines.Overall scores are 4.100 and 0.961; Daily Chat Spoken decreases from 4.016 to 3.965 while Rubric increases from 0.955 to 0.965.Source: §6.4 · pp. 13–14, Table 9 · p. 14
  • On FDB v1.5, response to background speech falls from 0.7300 to 0.1300; on Talking to Others, it falls from 0.1300 to 0.0300.On the in-house Qwen-Audio-FDB sets, session-level pass rates reach 0.9600 for multi-party interaction and 0.6800 for semantic control, versus 0.0800 and 0.0700 for 3.0.Source: §6.4 · pp. 15–16, Table 10 · p. 15, Table 11 · p. 15

6.5 Reliability and safety

  • 3.1 leads six of seven displayed safety and reliability metrics and exceeds GPT-Realtime-2 in nine of ten risk-category safety pass rates.Its multi-turn attack success rate falls by 54.50 percentage points in Chinese and 40.00 percentage points in English; adult content is the only risk category where its score is lower, by 0.90 points.Source: §6.5 · p. 16, Table 12 · p. 16, Table 13 · p. 17
  • In a 50-session human red-team study, five testers evaluated Qwen-Audio-3.1-Realtime, which achieved a 92.00% session-level safety pass rate versus 96.00% for GPT-Realtime-2; some safety failures remained.Source: §6.5 · p. 16

7 From Realtime Model to Persistent Voice Agent

The Voice Harness extends spoken interaction with durable background tasks while keeping dialogue, execution state, and stored information distinct. Its hybrid routing performs well on the in-house protocol, but the prototype uses a Qwen-Audio-3.0-Realtime foreground and does not evaluate the formal asynchronous lifecycle.

7.1 Foreground–background coordination

  • The Voice Harness pairs a real-time foreground for dialogue and direct tools with an orchestration runtime and background agent for durable multi-step work.The foreground submits a self-contained objective, constraints, and referenced inputs; background progress, questions, permissions, and results remain linked to the originating task.Source: §7 · p. 17, §7.1 · pp. 17–18, Figure 6 · p. 17

7.2 Task lifecycle and result delivery

  • Task acceptance, execution, completion, and result delivery are distinct events, so a finished artifact can wait for a suitable conversational moment.Speech interruption does not cancel work; cancellation requires an explicit control event and executor confirmation.Source: §7.2 · p. 18, Figure 7 · p. 18

7.3 System-level routing evaluation

  • The routing study uses a Qwen-Audio-3.0-Realtime foreground, and its simulated benchmark does not measure the formal asynchronous lifecycle.Latency is measured on a separate common-success subset conditioned on all four paths succeeding; the multi-step subset contains six turns.Source: §7.3 · pp. 18–19
  • 91.04% task success: hybrid routing exceeds foreground-only execution by 18.66 percentage points and delegating every audio request by 10.45 percentage points.On the separate latency subset, hybrid routing has the lowest overall mean at 4.729 seconds; all 65 short cases stay in the foreground and all six multi-step cases are delegated.Source: Table 14 · p. 19, §7.3 · p. 19

7.4 Memory and profile

  • Memory, reference knowledge, and live task state have separate roles, while user preferences and memories remain bounded and subordinate to current instructions and safety rules.The task ledger is the source of truth for changing execution status; profile learning updates attributes only after corroboration across multiple sessions.Source: §7.4 · pp. 19–20

8 Limitations and Safety

The report identifies uneven evaluation coverage and limits on attributing observed changes to individual training components. Persistent-agent evidence is also separate from evaluation of the released 3.1 model.

8 Limitations and Safety

  • The reported suite does not evaluate long-horizon executable spoken tasks or large-scale human red-teaming, and broader real-recording and deployment-condition coverage remains needed.EVA-A covers 200 of 213 sessions under a configuration that differs slightly from the official setup.Source: §8 · p. 20
  • Most comparisons are descriptive, and suite-wide confidence intervals, repeated-seed tests, and contamination analyses are unavailable.Controlled ablations are needed to attribute changes to Core-Cocktail SFT, either OPD path, environment evolution, or rollout granularity.Source: §8 · p. 20
  • Persistent-agent evidence comes from a separate Voice Harness with a Qwen-Audio-3.0-Realtime foreground, leaving end-to-end evaluation of released 3.1 and the formal asynchronous lifecycle open.Runtime memory, permission, and task-state mechanisms also require validation beyond learned model compliance.Source: §8 · p. 20
  • Future evaluation should add real recordings and deployment conditions, long-horizon executable tasks, larger human safety studies, and end-to-end latency and cost measurements.Source: §8 · p. 20

9 Conclusion

Qwen-Audio-3.1-Realtime brings spoken intelligence, executable actions, and conversational policy together through Think, Act, and Speak and Coordinate. Under the reported settings, it improves over 3.0 on several capability areas, while persistent-agent routing remains a separate Voice Harness study.

9 Conclusion

  • Think, Act, and Speak and Coordinate connect spoken intelligence, executable actions, and conversational policy; self-evolving environments and multi-granularity rollouts train task completion and policy compliance.Source: §9 · p. 20
  • Under the reported settings, 3.1 improves on 3.0 in audio instruction following, multilingual reasoning, speech-conditioned tool use, and selected conversational and full-duplex behaviors.The separate Voice Harness routing study uses a 3.0 foreground; further work should test the capabilities together in long-horizon spoken tasks with real recordings and end-to-end latency measurements.Source: §9 · p. 20

10 Contributions and Acknowledgments

The authors are listed alphabetically by last name.

10 Contributions and Acknowledgments

  • The author list begins with Lujia Bao, Qian Chen, Luyao Cheng, and Chong Deng.Source: §10 · p. 20
  • The middle of the author list includes Yuxiang Kong, Xiangang Li, Xu Li, Jiaqing Liu, and Chao-Hong Tan.Source: §10 · p. 20
  • The author list ends with Xilou Wang, Haoxiang Xu, Junhao Xu, Liang Yi, Binbin Zhang, Qinglin Zhang, and Qiquan Zhang.Source: §10 · p. 20

A Anatomy of an Executable Environment

The flight-search environment combines a business policy, executable tools, and a database to define task episodes. Its checks cover both state changes and dialogue behavior.

A Anatomy of an Executable Environment

  • The environment defines supported flight searches, pricing, airport lookup, and fare holds, while policy bars ticketing and payment and limits holds to flex fares and two active holds per customer.Expired holds do not count toward the limit, even when their stored status is active.Source: A Anatomy of an Executable Environment · p. 23
  • Tools operate on each episode’s private database copy, expose a schema derived from their signatures and docstrings, and return failures as data for recovery.The example place_hold tool locks the quoted fare for 72 hours and returns a hold reference.Source: A Anatomy of an Executable Environment · p. 23
  • Each episode starts from a fresh copy of the initial database, with current_date fixed for exact checks; the seed includes an active-status hold that is already expired.Counting holds by status alone would therefore give an incorrect total.Source: A Anatomy of an Executable Environment · p. 24
  • A task tuple specifies the initial database, simulated user and goal, expected outcome, required terminal-state values, reference tool calls, and dialogue assertions.The user script can include spoken-style cues, while assertions can require search before a hold, explicit confirmation, and truthful reporting afterward.Source: A Anatomy of an Executable Environment · pp. 24–25
  • Construction replays the reference solution against the expected state, while rollout checks evaluate the terminal-state subset, permitted write set, dialogue, and tool trace.A write outside the permitted set fails the episode.Source: A Anatomy of an Executable Environment · p. 25
Loading 2609.25176v2…