PaperlayerView exact revision on arXiv

Source-linked AI summary

StepAudio 3 Realtime Technical Report

Bin Lin, Bo Zhao, Boyang Zhang, Boyong Wu, Chao Yan, Chen Geng, Chen Wu, Cheng Yi, Chengli Feng, Chenglin Zhu, Chengting Feng, Chengyuan Yao, Daijiao Liu, DanNi Wan, Daxin Jiang, Dongjian Li, Dongqing Pang, Fei Tian, Feng Tian, Future Li, Gang Yu, Guanglong Yang, Haoyang Zhang, Hongyuan Wang, Jia Peng, Jiahao Song, Jialong Xue, Jiamin Fan, Jiangjie Zhen, Jianzheng Gao, Jincheng Wen, Jinghua Liang, Jinglan Gong, Jun Chen, Li Xie, Liang Zhao, Lifang Zhang, Lingli Ji, Lun Cai, Min Xu, Peilin Li, Peng Yang, Pengfei Tan, Qingjian Lin, Qinxin Du, Ruijie Xiong, Runze Li, Shenghua Hu, Shengqian Qin, Shi Qiu, Siqi Tu, Siyi Zhou, Tianjiao Deng, Wanying Lu, Weiming Niu, Wen Sun, WenWen Qu, Xiangyu Zhang, Xianwei Zhang, Xiaosu Su, Xing Chen, Xinyu Liu, Xuerui Yang, Yan Wu, Yang Li, Yang Yang, Yechang Huang, Yibo Zhu, Yifan Zhang, Yinuo Yan, Youjun Chen, Yu Fu, Yu Luo, Yu Zhou, Yujie Chen, Yumang Wang, Yunzhou Ju, Yuxiang Yang, Yuxin Li, Yuxin Zhang, Zekai Liu, Zengwei Yao, Zhaoxin Yuan, Zhenwei Mou, Zhiquan Zhang, Zhiyue Wu, Zichao Li, Zichao Zhou, Ziqi Ren, Zixuan Wang

arXiv:2609.14005v1cs.SDeess.AS

TL;DR

Realtime spoken interaction requires deep reasoning, low latency, fluid turn-taking, and tool use across ongoing exchanges. StepAudio 3 Realtime addresses these demands with a listen-converse-think-act loop combining Deep Perception, Seamless Duplex, Think-While-Speaking, and a Voice Agent. It reports broad benchmark strength, including 90.6 on MMSU, while identifying multi-turn constraint following and retail tool use as remaining gaps.

  • Problem

    Realtime spoken interaction must handle acoustic intent, pauses, backchannels, interruptions, deep reasoning, and tool tasks that may outlast the initiating exchange.

  • Method

    StepAudio 3 Realtime coordinates perception, conversational timing, reasoning, spoken delivery, and asynchronous tool execution through a continuous listen-converse-think-act loop.

  • Results

    The model shows broad audio-understanding strength, leads reported baselines on four of eight benchmarks, and reaches 90.6 on MMSU.

  • Takeaways & Limitations

    Think-While-Speaking supports spoken delivery before full reasoning completes, while the Voice Agent incorporates tool results into subsequent dialogue.

Abstract

from arXiv · show

Realtime spoken interaction demands deep reasoning, prompt responses, and fluid turn-taking. We present StepAudio 3 Realtime, an audio-language foundation model organized around a continuous listen-converse-think-act loop. Deep Perception captures rich acoustic cues to interpret user intent, while Seamless Duplex models synchronized audio streams to handle pauses, backchannels, and interruptions naturally. Crucially, we resolve the tension between deep deliberation and latency via Think-While-Speaking, executing private reasoning in parallel with spoken delivery. In reasoning mode, StepAudio 3 reaches a 73.0 macro average on StepAudioChat. With Think-While-Speaking, it achieves dialogue and reasoning performance comparable to dedicated reasoning models while speaking in real time. Furthermore, an integrated Voice Agent handles asynchronous tool execution without disrupting the dialogue flow. StepAudio 3 Realtime achieves top-tier performance across key dimensions: an exceptional 90.6 on the MMSU benchmark, 98.9 Overall on the Artificial Analysis Full-Duplex Bench, and a 56.0% macro task-success rate on $τ$-Voice.

1 Introduction

StepAudio 3 Realtime addresses the demands of spoken interaction by coordinating acoustic perception, conversational timing, reasoning, and tool use in a continuous loop. It reports broad benchmark strength while retaining gaps in multi-turn constraint following and retail tool use.

  • 1 Introduction: Natural spoken interaction requires handling incomplete pauses, acknowledgments, interruptions, deep reasoning, and tool tasks that may outlast the initiating exchange.
  • 1 Introduction: Streaming and full-duplex systems combine acoustic and linguistic information with overlapping listening and speaking to account for content, vocal delivery, and timing.
  • 1 Introduction: StepAudio 3 Realtime organizes perception, reasoning, and action as a listen-converse-think-act loop for spoken interaction.Deep Perception captures linguistic and nonverbal acoustic evidence, while a streaming Voice Agent carries intent into tool execution and returns results to dialogue.
  • 1 Introduction: The realtime model leads reported baselines on four of eight audio-understanding benchmarks and achieves the highest reported Overall score on the Artificial Analysis Full-Duplex Bench.The reported evaluation also identifies remaining gaps in multi-turn constraint following and retail tool-use tasks.

2 Realtime Conversational Loop

The realtime conversational loop maintains an evolving context containing speech, dialogue state, reasoning progress, and tool status. It uses this context to coordinate listening, speaking, reasoning, and asynchronous action as new speech or tool results arrive.

  • 2 Realtime Conversational Loop: User speech, model speech, and tool results can arrive while other interaction components remain in progress, so the system coordinates them through evolving conversational context.
  • 2 Realtime Conversational Loop: The context retains acoustic and linguistic evidence, dialogue history, the current turn, reasoning progress, and tool-execution status.Model-side speech helps interpret overlapping user utterances, while new speech and tool results can change subsequent decisions.
  • 2 Realtime Conversational Loop: Seamless Duplex handles pauses, backchannels, and substantive interruptions while Think-While-Speaking begins spoken delivery before the full reasoning trace is complete.Adaptive Thinking selects when explicit reasoning is useful, and multi-token prediction accelerates private reasoning.
  • 2 Realtime Conversational Loop: The Voice Agent resolves requests and arguments before execution, incorporates returned evidence into dialogue, and lets users continue speaking during ongoing tasks.

3 Model Architecture and Foundation Training

StepAudio 3 Realtime combines a large mixture-of-experts language model with an audio encoder, adapter, text pathway, and full-duplex audio loop. Its foundation training progresses from modality alignment through multimodal learning and cooldown, with expanded context and quality-aware audio data.

  • 3 Model Architecture and Foundation Training: The model uses approximately 196 billion total parameters and 11 billion active parameters per token, with Step 3.7 Flash as its language backbone and Qwen3-Omni’s AuT encoder as its audio frontend.
  • 3 Model Architecture and Foundation Training: Audio representations pass through an encoder and adapter into the LLM decoder alongside separate text input, while generated streaming audio returns to the model audio stream.User and model audio form the two full-duplex streams.
  • 3 Model Architecture and Foundation Training: The speech generator produces incremental output with context-appropriate tone, rhythm, pauses, and hesitation to connect response content with communicative intent.
  • 3 Model Architecture and Foundation Training: Automated audio curation filters, resegments, annotates, transcribes, cross-checks, and grades samples to support quality-aware sampling for sustained realtime perception and interaction.
  • 3 Model Architecture and Foundation Training: Pretraining uses modality alignment, multimodal mixed training, and cooldown over 1.2T tokens with a fixed 32K sequence length.
  • 3 Model Architecture and Foundation Training: Midtraining extends context to 128K and increases perception, synthetic-conversation, and voice-agent data for longer histories, requirements, tool results, planning, and spoken follow-up.

4 Deep Perception: Speech Recognition and Audio Understanding

The paper develops separate ASR specialization alongside broad audio understanding, using curated and context-rich training data. StepAudio 3 ASR Max leads the reported transcription benchmarks, while StepAudio 3 Realtime shows broad audio-understanding strength with specific multi-turn limitations.

  • 4 Deep Perception: Speech Recognition and Audio Understanding: ASR training combines short labeled utterances, long pseudo-labeled recordings, agreement filtering, context-aware examples, and targeted augmentation for rare or confusable terminology.
  • 4 Deep Perception: Speech Recognition and Audio Understanding: StepAudio 3 ASR Max achieves the best results on all four ContextASR-Bench subsets, with macro-average error rates of 5.67% for English and 1.23% for Mandarin.These results characterize the ASR-specialized model rather than the realtime model.
  • 4 Deep Perception: Speech Recognition and Audio Understanding: Audio-understanding examples are built by sampling and capability tagging recordings, constructing clip-specific questions, applying multi-model labeling, and filtering for quality and agreement.
  • 4 Deep Perception: Speech Recognition and Audio Understanding: StepAudio 3 Realtime leads reported baselines on four of eight audio-understanding benchmarks, including MMSU at 90.6 versus 83.6 and MMAR at 86.5 versus 81.7.It also leads on Step-Caption and MTalk-Bench, while trailing Gemini 3.1 Pro on AudioMultiChallenge by 17.7 points.
  • 4 Deep Perception: Speech Recognition and Audio Understanding: The realtime model shows broad strength in spoken-language understanding, audio-grounded reasoning, and nonverbal acoustic perception, while maintaining and revising constraints over natural multi-turn audio remains an improvement area.
  • 4 Deep Perception: Speech Recognition and Audio Understanding: Quality-controlled SFT data improve MMSU from 78.78 to 89.70 and MMAR from 74.70 to 84.50 despite using roughly one twentieth as many examples as random sampling.WildSpeech and the MTalk-Bench macro average also improve under quality-controlled data.

5 Seamless Duplex: Conversational Floor Management

Seamless Duplex manages the conversational floor by combining acoustic timing, semantic completeness, and dialogue context. It distinguishes pauses, turn endings, acknowledgments, and interruptions to decide whether to listen, respond, continue speaking, or yield.

  • 5 Seamless Duplex: Conversational Floor Management: It interprets brief acknowledgments and substantive interruptions differently, guiding whether model speech continues or yields the floor.The interpretation depends on the utterance’s relation to the ongoing dialogue.
  • 5 Seamless Duplex: Conversational Floor Management: Seamless Duplex integrates user speech, model speech, and dialogue history for context-aware conversational-floor management.The architecture uses these signals to interpret interaction as the dialogue unfolds.

A Dual-stream recurrent architecture

The dual-stream architecture represents audio and interaction states on a shared timeline while using both speech streams and dialogue history for floor management. It is trained for streaming interaction behaviors and achieves leading full-duplex evaluation results.

  • 5.1 Streaming Interaction States: Audio is organized into 320 ms blocks, each followed by a state or text token in the time-interleaved representation.This representation tracks the interaction state incrementally.
  • 5.1 Streaming Interaction States: The model uses acoustic evidence, semantic context, both audio streams, and dialogue history to decide when to listen, speak, respond, or yield.Model-side speech helps interpret overlapping user utterances in relation to what the user is hearing.
  • 5.1 Streaming Interaction States: Training combines streaming ASR, VAD, utterance-completeness prediction, and interaction data covering turn taking, backchannels, interruptions, and background speech rejection.Midtraining includes over 10,000 hours of synthetic full-duplex interaction data.
  • 5.1 Streaming Interaction States: 98.9 Overall ranks StepAudio 3 Realtime first on the Artificial Analysis full-duplex evaluation, above Qwen Audio 3.0 Realtime Plus at 98.4.The benchmark covers pause handling, turn taking, user interruption handling, and backchannel handling.
  • 5.4 Evaluation: Category scores reach 100.0 for turn taking, 99.0 for user interruption handling, 98.9 for pause handling, and 98.0 for backchannel handling.These results indicate balanced control over when to listen, speak, and yield across complementary interaction behaviors.

6 Conversational Intelligence and Realtime Reasoning

StepAudio 3 Realtime trains conversational intelligence across diverse multi-turn contexts and pairs dialogue reasoning with Think-While-Speaking. In reasoning mode, it achieves a 73.0 macro average on StepAudioChat, below Kimi K3 but above two listed baselines.

  • 6 Conversational Intelligence and Realtime Reasoning: Think-While-Speaking coordinates spoken responses with ongoing private reasoning, while Adaptive Thinking selects explicit reasoning and MTP accelerates it.The design targets different deliberation budgets for routine and complex conversational turns.
  • 6.1 StepAudioChat Benchmark: StepAudioChat evaluates dialogue behavior and expressed reasoning on eight capability dimensions using independently checkable criteria and valid and flawed response controls.Structural, semantic, automated, and targeted human checks filter retained examples and support judge calibration.
  • 6.2 Multi-Turn Dialogue Data: The training data target intent and constraint following across turns, clarification of underspecified requests, and context-adapted responses.Examples span diverse topics, personas, interaction lengths, and multiple valid answer phrasings.
  • 6.2 Multi-Turn Dialogue Data: Multi-turn examples vary topics, personas, turn depth, and target capabilities, with later turns refining constraints, resolving ambiguity, revisiting evidence, or changing direction.Quality control reviews context-query linkage, capability satisfaction, response quality, persona consistency, and spoken naturalness.
  • 6.2 Multi-Turn Dialogue Data: 73.0 macro average places StepAudio 3 above Doubao 2.0 Lite at 70.5 and DeepSeek-V4-Flash at 71.4, but below Kimi K3 at 77.1.It ranks second on reasoning, memory, knowledge, conversational pragmatics, and persona and role consistency.
  • 6.2 Multi-Turn Dialogue Data: Strong aggregate reasoning does not imply uniformly stronger instruction following or persona and role consistency.The reported category differences separate overall reasoning strength from consistently stronger behavior across capabilities.

6.3 Think-While-Speaking

Think-While-Speaking runs private reasoning and spoken delivery concurrently, with Adaptive Thinking selecting when to deliberate and MTP accelerating the private stream. Evaluations show gains in selected capabilities but also expose selection and efficiency limitations.

  • 6.3 Think-While-Speaking: Two concurrent calls act as a Formulation Brain for private reasoning and an Articulation Brain for response segments conditioned on reasoning available so far.Playback-aware scheduling releases segments while formulation continues, and later continuation can use the completed reasoning state.
  • 6.3.1 Adaptive Thinking: Adaptive Thinking invokes explicit reasoning at rates from 51.5% to 82.0% across eight categories.Full thinking gains most over forced no-think in Reasoning by 11.37 points, followed by Persona and Role Consistency by 8.37 points and Knowledge by 5.94 points.
  • 6.3.1 Adaptive Thinking: Relative to Direct SFT, Adaptive Thinking improves Dialogue Pragmatics from 63.59 to 65.87 but reduces Reasoning from 71.89 to 66.80.Adaptive Thinking is trained separately with reasoning-selection supervision.
  • 6.3.1 Adaptive Thinking: Reasoning has a 59.5% think rate despite benefiting most from full thinking, whereas Faithfulness has a 79.2% rate but gains only 1.72 points.These aggregate comparisons do not establish the optimal reasoning decision for individual turns.
  • 6.3.2 Accelerating Private Reasoning with MTP: MTP configurations improve Reasoning and Memory over baseline, while Instruction Following declines.MTP uses draft-token prediction to reduce sequential target-model steps in the private thinking stream.
  • 6.3.2 Accelerating Private Reasoning with MTP: MTP5 accepts 2.153 draft tokens per target-model step versus 1.801 for MTP3, while later draft heads provide diminishing gains.The fourth and fifth heads have strict marginal acceptance rates of 10.7% and 5.7%, respectively.
  • 6.3.2 Accelerating Private Reasoning with MTP: Acceptance alone does not determine net efficiency because reasoning length and decoding costs also affect wall-clock performance.Strict verification for spoken output does not eliminate errors from incomplete private reasoning.

6.4 Model Merging for Capability Integration

The model merges specialized teachers through weighted parameter averaging to combine complementary capabilities without adding inference-time components. The merge improves overall balance across audio, text, and dialogue, though it does not uniformly surpass every specialist.

  • Weighted parameter merging: Four specialized teachers are combined by directly averaging their parameters with a normalized 3:1:1:1 weighting selected using held-out evaluations.Parameter-space integration adds neither model components nor inference-time routing.
  • Capability integration: The merging strategy preserves complementary strengths and enables independently developed data mixtures to be recombined without retraining on their full union.Evaluation therefore considers both capability-specific scores and overall balance.
  • Model merging results: 81.3 audio-understanding macro average ties the best teacher, while 76.5 on general text is the highest macro average and 73.0 on dialogue exceeds three teachers but trails the strongest at 74.2.The merge leads on MMAU, WildSpeech, HMMT 2026 Feb, and GPQA Diamond, and ties on MultiChallenge.

7 Full-Duplex Voice Agent

The full-duplex voice agent routes requests among direct responses, lightweight tools, and asynchronous backend execution while maintaining conversational context. It achieves competitive end-to-end task completion, with strongest relative performance in telecom and remaining weakness in retail.

  • Voice-agent execution: The agent selects direct responses, lightweight tools, or asynchronous backend execution based on request requirements.Long-running backend tasks continue while users request updates, add requirements, or change topics.
  • Voice-agent execution: Before consequential external actions, the model gathers missing information through clarification or context retrieval and obtains required confirmation.Tool and backend results are incorporated into later conversational responses.
  • Training: Training combines targeted voice-agent dialogues with filtered multi-step trajectories covering routing, clarification, evidence grounding, argument consistency, and spoken interaction.The training also discourages unnecessary tool invocation and unsupported claims about private information or completed work.
  • Evaluation: 56.0% macro task success is close to Grok’s 56.5% and above Qwen’s 54.6% and GPT’s 45.7%.The benchmark covers airline, retail, and telecom customer-service tasks under interruptions, backchannels, and background noise.
  • Evaluation: 70.2% telecom success is the highest among evaluated models, while the 60.0% airline score is within 2.0 points of the best reported 62.0%.Retail remains weaker at 37.7%, compared with Grok’s 49.7%.

8 Evaluation

The evaluation spans speech recognition, audio understanding, dialogue and reasoning, full-duplex interaction, agentic completion, and general text. StepAudio 3 Realtime shows broad capability and strong interaction performance, while results vary across reasoning and tool-use tasks.

  • Evaluation design: The evaluation compares six domains using task-specific benchmarks and domain-specific baselines, with higher scores preferred except for ASR error rates.The domains progress from recognizing speech to understanding context, managing the conversational floor, and completing external tasks.
  • Evaluation design: τ-Voice measures tool-grounded customer-service completion across airline, retail, and telecom, while Full Duplex Bench measures pauses, turn taking, interruptions, and backchannels.A τ-Voice task succeeds when the final database state matches its target.
  • Results: StepAudio 3 Realtime leads four of eight audio-understanding benchmarks and reaches an 81.3 macro average, within 0.5 points of Gemini 3.1 Pro.Its clearest gains appear on MMSU and MMAR, with additional strength on speaker attributes, paralinguistic cues, ambient sound, and multi-turn audio context.
  • Results: 73.0 is the reasoning-mode StepAudioChat macro average, while realtime Think-While-Speaking reaches 70.41, comparable to dedicated reasoning baselines at 70.5 and 71.4.The realtime result indicates deliberation can occur during speech rather than producing only low-latency surface responses.
  • Results: The model leads HMMT at 86.8, but GPQA Diamond and MultiChallenge remain below Gemini 3 Flash, and retail task completion remains weaker than Grok’s.These results show strong but non-uniform performance across text and tool-use tasks.

9 Conclusion

StepAudio 3 Realtime integrates perception, conversational timing, reasoning, and tool use into continuous spoken interaction. Its evaluations show strong audio understanding and floor management alongside uneven reasoning and task-execution performance.

  • Conclusion: Continuous interaction lets deliberation and external tasks proceed alongside conversation, with new user input and returned evidence informing later responses.The conclusion identifies more effective reasoning-effort allocation and more reliable extended task execution as future needs.
  • Conclusion: Adaptive Thinking reduces explicit reasoning frequency but has uneven effects on answer quality, while multi-turn constraint handling and retail completion remain improvement areas.The conclusion therefore qualifies the model’s strong interaction results with variation across reasoning and tool-use tasks.
Loading 2609.14005v1…