PaperlayerView exact revision on arXiv

Source-linked AI summary

Audio Interaction Model

Zhifei Xie, Zihang Liu, Ze An, Xiaobin Hu, Yue Liao, Ziyang Ma, Dongchao Yang, Mingbao Lin, Deheng Ye, Shuicheng Yan, Chunyan Miao

arXiv:2606.05121v1cs.SDcs.AIcs.CLcs.MMeess.AS

TL;DR

Existing LALMs are offline and existing streaming models are task-specific, leaving a gap for general real-time audio interaction. The paper introduces Audio-Interaction and SoundFlow to unify streaming perception, response decisions, and generation, while retaining offline capabilities. Across the reported evaluations, the model preserves competitive mainstream audio performance and adds streaming and proactive-intervention capabilities.

  • Problem

    Existing LALMs require complete audio clips, while streaming systems are task-specific and lack a general streaming audio language model.

  • Method

    Audio-Interaction uses SoundFlow’s always-on perceive–decide–respond loop with streaming-native data, comprehension-aware training, and asynchronous inference.

  • Results

    Audio-Interaction preserves competitive mainstream audio performance while unlocking streaming capabilities inaccessible to offline LALMs.

  • Takeaways & Limitations

    The paper presents a unified model for conventional offline and streaming tasks, general streaming audio instruction following, and proactive audio intervention.

Abstract

from arXiv · show

Audio is an inherently interactive modality, yet today's Large Audio Language Models (LALMs) are offline, and streaming audio models each handle only a single task such as streaming ASR or voice chatting. It is time to unify them into one online LALM: a model that, through an always-on perceive-decide-respond loop, listens to sound, environment, and instructions in real time and reacts on the fly. We formalize this regime as the Audio Interaction Model, and realize it with Audio-Interaction, a unified streaming model that retains offline task execution while adding online general audio instruction following, from dialogue to full voice chatting, deciding when to respond from the semantics of the stream. To enable this, we propose SoundFlow, a framework that instantiates the perceive-decide-respond loop end to end, from data to training to deployment, through streaming-native data construction, comprehension-aware training, and asynchronous low-latency inference for stable real-time interaction. We further construct StreamAudio-2M, a 2.6M-item streaming corpus spanning 7 fundamental abilities and 28 sub-tasks, and Proactive-Sound-Bench for evaluating proactive audio intervention. Across 8 benchmarks, Audio-Interaction preserves competitive performance on mainstream audio tasks while unlocking capabilities inaccessible to offline LALMs, including real-time ASR, streaming audio instruction following, and proactive help.

1 Introduction

Audio-Interaction addresses the mismatch between offline LALMs and audio’s continuous, interactive nature by processing streams incrementally and deciding when to respond. SoundFlow supports this unified online model through comprehension-grounded triggering and end-to-end streaming data, training, and inference.

  • Research gap: Current LALMs wait for complete audio, while task-specific streaming systems require separate models for narrow capabilities.The paper identifies both offline processing and isolated streaming tasks as limitations of existing approaches.
  • Research challenges: Interactive audio requires deciding at each chunk whether to respond based on semantic understanding of the unfolding context.The decision is difficult because intervention supervision is sparse and temporally ambiguous.
  • Proposed approach: AUDIO-INTERACTION consumes audio chunk by chunk and forms an always-on perceive–decide–respond loop that unifies translation, recognition, dialogue, and other capabilities.The model can choose between responding and remaining silent while processing the stream.
  • Proposed approach: SoundFlow provides streaming-native data synthesis, comprehension-aware training, and asynchronous inference for real-time interaction.Its components address coherent interaction data, context forgetting and false triggering, and low-latency encoding-decoding coordination.
  • Empirical validation: 58.15 vs. 57.81 on MMAU shows that interactive conversion preserves competitive mainstream audio performance.The authors also report stronger performance in several full-speech and multi-turn settings.

2 Related Work

Prior work largely separates offline LALMs from narrow or turn-based streaming systems. Existing models either require complete clips before responding or support limited functions such as dialogue or transcription.

  • Offline audio language models: Offline LALMs typically combine an audio encoder, adapter, and language model, but require the complete audio clip before responding.This design follows a fixed offline input-output formulation.
  • Streaming multimodal systems: Streaming dialogue models ingest audio chunk by chunk but generally respond only after an utterance ends.They therefore remain turn-based rather than continuously interactive.
  • Streaming multimodal systems: Fully streaming systems such as Moshi treat non-speech events as background, while streaming ASR is limited to transcription.These examples illustrate the narrow scope of existing streaming capabilities.

3 Audio-Interaction

Audio-Interaction converts offline audio understanding into an always-on streaming loop that consumes audio chunks, decides whether to respond, and generates responses in real time. SoundFlow supports this through streaming-native data construction, comprehension-aware training, and asynchronous inference.

  • Overview: Audio-Interaction incrementally consumes audio chunks and autonomously chooses between remaining silent and responding.The model unifies perception, intervention decisions, and response generation in a continuous streaming process.
  • Streaming Training: SoundFlow organizes audio, intermediate representations, and supervision into a unified temporal sequence while jointly training language modeling and response triggering.This training design supports semantic decisions about when to respond or remain silent across real-time tasks.
  • Streaming Training: The streaming model processes fixed-length 400 ms chunks and predicts a <silent> or <response> token at each step.Silence continues listening; response switches the model into autoregressive generation.
  • Streaming Training: History review training addresses insufficient context retention by inserting questions about earlier content into later sequence positions.The intervention explicitly encourages long-range contextual retrieval in long streaming sequences.
  • Streaming Training: A dedicated streaming objective is optimized alongside language modeling to train the special streaming control token.The overall objective combines target text-token prediction with streaming-control prediction, weighted by λ.
  • Asynchronous Inference: FIFO asynchronous inference continuously queues encoded audio chunks and triggers decoding when the decoder is not speaking.The design eliminates inference stalling and reduces first-frame latency for resuming listening after responses by 4.5×.

4 StreamAudio-2M Dataset

StreamAudio-2M is a streaming-native corpus designed for continuous, context-dependent audio interaction rather than short isolated triplets. It spans broad streaming capabilities and supports evaluation of proactive intervention.

  • Overview: StreamAudio-2M contains 2.6M items totaling 302k hours, with each sample comprising 3–15 turns of heterogeneous interaction.Samples interleave events with sparse, context-dependent response cues.
  • Overview: The corpus covers 7 major categories and 28 streaming sub-tasks, including proactive response, voice chatting, real-time ASR, and streaming translation.Its categories span agentic, conversational, understanding, music, recognition, and translation abilities.
  • Curation Pipeline: Data collection combines dialogue, ASR, speech-translation, music, and audio-question-answering sources with acoustic events and environmental noise.The foundational sources contribute approximately 1.64M items and 8,900 hours, with additional acoustic-event clips and noise sources for environmental conditioning.
  • Evaluation: ProactiveSound-Bench evaluates whether models trigger or abstain correctly across 644 human-designed acoustic events.Events span 6 top-level categories and 17 sub-categories, with Single and Multiple tiers testing isolated and sustained decisions.

5 Experiments

Experiments show that AUDIO-INTERACTION retains competitive mainstream audio performance while adding streaming, proactive, and real-time capabilities. Analyses attribute these results to streaming representations, control decisions, asynchronous inference, data construction, and balanced training objectives.

  • Main Results: AUDIO-INTERACTION reaches 58.15 on MMAU and preserves competitive performance across general audio, dialogue, recognition, translation, and proactive-response benchmarks.The evaluation covers eight audio benchmarks and compares against audio LLMs, omni LLMs, and task-specialized models.
  • Main Results: CoVoST2 improves over initialization by +15.72/+17.04 BLEU on en-zh/zh-en, while LibriSpeech incurs only a marginal WER regression from chunk-wise streaming decoding.The model also matches or exceeds its base model on three of four dialogue benchmarks.
  • Model Analysis: Continuity is reconstructed at the earliest decoder layer through cross-chunk KV-cache access across four tasks, rather than by task-specific heads.The continuity ratio compares boundary-pair cosine similarity with intra-chunk similarity, where 1.0 denotes seamless continuity.
  • Model Analysis: A single attention head, L35H14, dominates streaming-control-token generation across four tasks; ablating it reduces S2TT token-match score by 0.88.The analysis ablates each of 576 attention heads individually.
  • Ablation Study: FIFO scheduling reduces average first-chunk latency from 831 ms to 392 ms and stall rate from 5.2% to 0.0%, enabling stable low-latency inference.The comparison reports a 2.12× slowdown without FIFO scheduling.
  • Ablation Study: Streaming SFT, TFJP preprocessing, and hierarchical event selection improve triggering, with full AUDIO-INTERACTION achieving 96.7% trigger accuracy.Removing TFJP preprocessing or hierarchical event selection lowers trigger accuracy by 7.1 and 3.9 points, respectively.
  • Ablation Study: The chosen 0.4 s chunk reaches 4.28 Alpaca and 58.2 MMAU at 392 ms latency, balancing accuracy and responsiveness better than tested alternatives.Smaller chunks lack semantic context, while 0.6 s and 0.8 s increase latency to 674 ms and 786 ms.

6 Conclusion

The conclusion identifies the gap between offline LALMs and continuous audio interaction and presents AUDIO-INTERACTION as a unified streaming model addressing that gap. Real-world recordings retain most synthetic-stream behavior, with degradation tied to acoustic difficulty rather than systemic failure.

  • Conclusion: AUDIO-INTERACTION formalizes the Audio Interaction Model as one system for offline tasks, streaming tasks, and general streaming audio instruction following.The conclusion positions this as a response to streaming models being confined to isolated tasks.
  • Real-World Validation: Across four natural-recording scenarios, trigger accuracy averages 58.9% versus 62.0% on a matched synthetic split, with scenario-specific acoustic degradation.Travel and Commute show the largest declines under crowd ambience and non-stationary noise.
  • Real-World Validation: Average first-chunk latency remains within ±25 ms of the synthetic measurement in every scenario, indicating robustness to recording-side jitter and device variation.Per-chunk silence rates correlate 0.91 with the matched synthetic split.
  • Real-World Validation: On real recordings, ablating L35H14 reduces token-match by 0.86 versus 0.88 on synthetic audio, preserving the model’s internal streaming decision pattern.This supports consistency of the dominant streaming-control mechanism across synthetic and naturally recorded audio.

A.2 Case Study

The case studies illustrate always-on interaction in household and office streams, where AUDIO-INTERACTION processes 0.4-second chunks and decides whether to remain silent or speak. The appendix also details the framework’s preprocessing, curation, supervision, and asynchronous inference components.

  • Home: The Home case is a continuous 30-second household stream in which AUDIO-INTERACTION listens every 0.4 s and exercises five of seven StreamAudio-2M task categories.The model decides at each chunk whether to remain silent or speak.
  • Office: The Office case is a continuous 60-second workday stream processed every 0.4 s and covering five of seven StreamAudio-2M task categories.The same silent-or-speak decision loop is used in the office scenario.
  • Framework Details: The appendix fixes the streaming chunk size at c = 400 ms and identifies fenc, fproj, and fdec as the inherited encoder, adapter, and language-model components.Optimization hyperparameters are deferred to Appendix E.
  • Framework Details: TFJP stabilizes clip audio before stitching through silence cutting, noise profiling, denoising, and energy- or spectral-entropy-based core-span selection.The module applies six operators sharing one STFT representation.
  • Framework Details: Hierarchical curation plans coherent scenarios, refines events into retrieval or generation paths, and verifies both sources with the same acceptance criteria.This process addresses contextual consistency when composing long-form streams.
  • Framework Details: Each streaming sample supervises one silent-or-response control token per chunk and language-model tokens only after a response trigger.Audio-encoder positions and the instruction prefix are masked from both objectives.
  • Framework Details: History-review and silent-audio prompts target long-context retention and false triggering, respectively, by generating context-dependent questions and auditing non-speech segments.The two failure modes receive dedicated comprehension-aware supervision.
  • Framework Details: The FIFO scheduler separates encoding and decoding through a shared queue, flushing accumulated features at interruption points while preserving autoregressive response generation.Drain-on-trigger keeps acoustic context aligned with wall-clock time after long responses.

B.4 Dataset Curation Pipeline

The dataset curation pipeline converts heterogeneous sources into coherent, verified streaming audio with explicit response timelines. It combines spoken-form rewriting, event planning and grounding, comprehension-aware prompts, and dual-track token supervision with asynchronous decoding procedures.

  • Spoken-Style Conversion: Text-form sources are rewritten for natural speech, rendered with CosyVoice, and accepted only when downstream ASR stays within τwer = 0.10, with up to two retries.A voice is sampled once per dialogue from a multi-voice pool.
  • Stream Composition: Validated clips are composed into long streams by sequentially concatenating foreground events, mixing background and ambient events, and adding two tiled noise tracks.Foreground, background, and ambient clips use role-dependent gains of 0 dB, −6 dB, and −12 dB.
  • Supervision: Streaming tokenization emits one <silent> or <response> target per chunk and supervises response text and <eos> only after a response trigger.Instruction and encoder-feature positions are masked from both target streams.
  • Inference: Asynchronous inference runs encoder and decoder loops concurrently, queues projected features, flushes them at interruption points, and performs autoregressive steps during responses.This FIFO procedure is formalized as Algorithm 3 and keeps decoding aligned with incoming audio.
  • Event Curation: Hierarchical event curation plans coherent scenes, refines sub-events into retrieval and generation candidates, and applies source-agnostic grounding verification.The same verifier is used for retrieved and synthesized clips.
  • Supervision: Comprehension-aware supervision generates delayed history-dependent questions and audits silent audio so irrelevant sounds do not trigger responses.The prompt templates use JSON-mode decoding and discard borderline silent-audio cases rather than labeling them.

C StreamAudio-2M Dataset Sources

StreamAudio-2M combines established public corpora into a streaming corpus covering speech, acoustic events, and environmental noise. Its sources are selected to support distinct streaming capabilities while improving reproducibility and robustness to real acoustic conditions.

  • Source design: StreamAudio-2M combines publicly available corpora selected for distinct capability slots in the streaming regime.The dataset favors established sources over scraped or proprietary collections because streaming composition already transforms each upstream signal.
  • Source accounting: Table 9 records each source corpus’s upstream item count and raw audio duration, with environmental-only sources marked separately.The table distinguishes source contributions before streaming composition from sources used only for environmental conditioning.
  • Speech-centric sources: Speech-centric sources support spoken dialogue, streaming ASR, speech-to-text translation, and audio question answering.MOSS supplies 392k multi-turn instances rendered into 4,900 hours of speech with multi-voice CosyVoice.
  • Acoustic event sources: Acoustic event sources combine real and synthetic data to cover common, rare, and context-specific events without excessive head-class bias.AudioSet supplies broadly sampled real events, while synthetic sources address sparse coverage for target ontology nodes.
  • Noise sources: Background noise is overlaid on every long-form stream so the model learns to suppress responses to non-foreground sound in non-silent environments.Three established noise corpora provide complementary acoustic conditions.

D.1 Task Definition

ProactiveSound-Bench evaluates whether an audio system should intervene and, when appropriate, generate a natural-language response. Its task boundary distinguishes safety-relevant or assistance-worthy sounds from benign everyday signals that should elicit silence.

  • Task definition: ProactiveSound-Bench jointly evaluates response triggering and natural-language response generation from audio input.The benchmark requires both deciding whether to respond and generating the response when triggered.
  • Response boundary: The benchmark requires intervention for illness or discomfort, severe weather, potential equipment damage, and hazardous environmental signals.Normal physiological sounds, routine equipment operation, and similar signals should not trigger responses.
  • Task distinction: Unlike sound event detection and audio captioning, ProactiveSound-Bench evaluates whether to intervene and what to say, not only acoustic recognition or description.SED typically outputs labels or temporal boundaries, whereas captioning produces neutral descriptions.
  • Taxonomy: The taxonomy covers human sounds, nature and environment, equipment, traffic, and music to represent everyday assistant-device scenarios.Its categories progressively partition sounds by their relation to the human body and by environmental or engineered sources.
  • Taxonomy: Table 10 provides conceptual meso-level category definitions, while exemplars are reported separately.

E Experiments Details

The experiments situate Audio-Interaction within both unified audio-language modeling and streaming-system research. The paper contrasts its single-model direction with task-specific streaming systems and offline general-purpose audio models.

  • Experiment configuration: Table 11 lists method, data, and optimization hyperparameters fixed or varied across the four-stage training recipe.Method and data constants follow ablation or recipe choices, while optimization settings vary by stage scale and trainable-parameter footprint.
  • Experiment configuration: Table 11 is the configuration reference for Audio-Interaction’s parameters.
  • Streaming systems: Streaming audio research includes dedicated systems for speech recognition, speech translation, and full-duplex spoken dialogue.These systems specialize in particular streaming functions rather than providing one unified model.
  • Audio large models: Audio large models provide broad audio capabilities but are described as a foundation for downstream tasks rather than a complete real-time interaction system.The related work positions them as general-purpose models for audio-based tasks.
  • Streaming AI systems: The paper contrasts its end-to-end model with cascaded streaming systems that coordinate several specialized components around text processing.Audio-Interaction aims to realize the capability within a single end-to-end model.

G Error Analyses

The error analyses identify recurring failures across recognition, reasoning, generation, and proactive intervention. Hallucination and irrelevant responses dominate several language-oriented tasks, while proactive systems struggle with both overreaction and missed safety signals.

  • Speech recognition: 60.2% of analyzed LibriSpeech errors are local token deviations, while 21.4% involve rare words and long utterances.The analysis groups phonetically or orthographically motivated substitutions with minor insertions and deletions in the largest category.
  • Speech translation: 82% of low-BLEU CoVoST2 English-to-Chinese cases are semantic hallucinations unrelated to the source audio.The remaining 18% are incomplete or mixed-language outputs containing untranslated English fragments.
  • Audio understanding: Approximately 20% of MMAU errors arise from generation collapse, with the remainder reflecting recognition or reasoning mistakes.Examples include confusion between acoustically similar sources, speaker-attribute misclassification, and incorrect categories despite partial reasoning.
  • Spoken question answering: Factual hallucinations account for 56.8% of LlamaQA errors and approximately 71% of WebQuestions errors.Other errors include temporal or quantitative mistakes and irrelevant or generalized responses.
  • VoiceBench: 53.5% of AlpacaEval-full errors are hallucinations and 46.4% are irrelevant responses or inappropriate refusals; SD-QA hallucinations comprise roughly 63%.The analysis also identifies miscomprehension and over-refusal in VoiceBench.
  • Proactive intervention: 59.8% of ProactiveSound-Bench errors are false positives, while 40.2% are false negatives concentrated in safety-critical domains.False positives include overreactions to benign daily sounds, whereas false negatives include missed traffic alarms and natural hazards.
Loading 2606.05121v1…