PaperlayerView exact revision on arXiv

Source-linked AI summary

MOSS-Audio Technical Report

Chen Yang, Chufan Yu, Hanfu Chen, Jie Zhu, Jingqi Chen, Ke Chen, Wenxuan Wang, Yang Wang, Yaozhou Jiang, Yi Jiang, Zhengyuan Lin, Ziqi Chen, Zhaoye Fei, Chenghao Liu, Donghua Yu, Jun Zhan, Kang Yu, Kexin Huang, Liwei Fan, Mingshu Chen, Qinyuan Cheng, Ruixiao Li, Shimin Li, Songlin Wang, Xingjian Zhao, Yang Gao, Yitian Gong, Yiyang Zhang, Zhe Xu, Xipeng Qiu

arXiv:2606.01802v3cs.SDcs.AI

TL;DR

MOSS-Audio addresses the need for one model that understands speech, environmental sound, music, and temporal audio events beyond transcription alone. It combines a dedicated audio-language architecture, multi-level feature injection, explicit time markers, and event-preserving branched supervision. The resulting Instruct and Thinking variants show strong performance across general audio understanding, speech captioning, ASR, and timestamped ASR, supporting their use as audio-understanding foundations for future voice agents.

  • Problem

    Existing audio-language systems must handle heterogeneous acoustic attributes and temporal events that narrow ASR-oriented frontends may discard.

  • Method

    MOSS-Audio combines a dedicated audio encoder, DeepStack cross-layer injection, explicit time markers, branched event-preserving annotation, and unified caption training.

  • Results

    MOSS-Audio shows strong performance across general audio understanding, speech captioning, ASR, and timestamped ASR, with Thinking strongest on broad understanding and Instruct strongest on speech-oriented tasks.

  • Takeaways & Limitations

    A single model family can cover descriptive understanding, transcription, temporal grounding, and reasoning over heterogeneous audio within one interface.

Abstract

from arXiv · show

MOSS-Audio is a unified audio-language model for speech, environmental sound, and music understanding, supporting audio captioning, time-aware question answering, timestamped transcription, and audio-grounded reasoning. MOSS-Audio couples a dedicated audio encoder with a modality adapter and a large language model: the encoder produces 12.5 Hz temporal representations, the adapter projects them into the decoder space, and the decoder generates autoregressive text outputs. Two design choices are central to the system: DeepStack cross-layer feature injection, which exposes the decoder to acoustic information from multiple encoder depths, and time markers, which provide explicit temporal cues by inserting timestamp markers into the audio-token stream. At the data level, we design an event-preserving audio annotation pipeline that segments raw audio at coherent event boundaries, applies branch-specific annotation to speech, music, and general audio, and merges the results into unified captions for pretraining. The intermediate branch-specific captions are further retained to support the construction of task-oriented SFT data. The model is pretrained on large-scale audio-language data, with time-aware objectives incorporated to support temporal grounding, and then undergoes multi-stage post-training to enhance instruction following and audio-grounded reasoning. We release 4B and 8B variants in both Instruct and Thinking configurations. MOSS-Audio achieves strong performance across general audio understanding, speech captioning, ASR, and timestamped ASR, positioning it as a promising understanding foundation for future voice agents.

1 Introduction

MOSS-Audio targets unified understanding of heterogeneous, temporal audio rather than transcription alone. It combines a shared audio-language framework, multi-level acoustic features, explicit time cues, broad annotation, and distinct Instruct and Thinking variants.

  • Motivation: Heterogeneous audio tasks require fine-grained acoustic, prosodic, environmental, musical, semantic, and temporal information.ASR, captioning, environmental sound understanding, and reasoning depend on different abstraction levels and often require determining when events occur.
  • Unified Model: MOSS-Audio unifies ASR, captioning, timestamped transcription, time-aware QA, and audio-grounded reasoning within one autoregressive text-generation framework.The model accepts audio and an instruction, then generates task-specific text while sharing the audio representation and language-decoding interface.
  • Unified Model: The model uses a 12.5 Hz audio encoder, modality adapter, and language model decoder to generate autoregressive text conditioned on audio and instructions.This encoder–adapter–decoder design combines broad acoustic processing with language-model instruction following and generation.
  • Architectural Choices: DeepStack injects features from multiple encoder depths, while explicit time markers make temporal information part of generation for timestamped and time-aware tasks.Together, these choices preserve acoustic evidence at multiple granularities and expose elapsed-time cues directly to the decoder.
  • Data and Training: The data pipeline preserves acoustic events, applies speech-, music-, and general-audio annotation, and merges heterogeneous supervision into unified captions.Intermediate branch-specific annotations also support task-oriented instruction data construction.
  • Results: MOSS-Audio provides 4B and 8B Instruct and Thinking variants with complementary strengths across direct execution and reasoning-oriented audio understanding.The report describes strong performance across general audio understanding, speech captioning, ASR, and timestamped ASR.

2 Architecture

MOSS-Audio is designed around a dedicated audio encoder and decoder interface that preserve broad acoustic information. DeepStack supplies multi-level encoder evidence, while time markers provide explicit temporal context for the language model.

  • 2.1 Overview: Existing ASR-oriented frontends often discard speaker, prosodic, environmental, and musical information needed for unified audio understanding.MOSS-Audio therefore trains a dedicated audio encoder for broad acoustic attributes and language-model alignment.
  • 2.1 Overview: The architecture contains a dedicated audio encoder, two GatedMLP cross-modal adapters, and a language-model decoder.The encoder converts log-mel features into temporal representations, while adapters project audio features into decoder space.
  • 2.1 Overview: The encoder produces efficient 12.5 Hz representations after 8× temporal downsampling of 128-channel log-mel spectrograms.Its approximately 0.6B-parameter module uses three stride-2 Conv2D layers followed by a 32-layer Transformer.
  • 2.3 DeepStack Cross-Layer Feature Injection: A final encoder layer alone cannot capture all required granularities because lower and intermediate layers retain local acoustic, prosodic, transient, and speaker cues.This motivates combining information across encoder depths rather than relying only on the final representation.
  • 2.3 DeepStack Cross-Layer Feature Injection: DeepStack projects intermediate encoder states through a merge adapter and injects them into selected early decoder layers.The mechanism exposes multiple abstraction levels to language-model decoding while retaining the final encoder output through the primary adapter.
  • Temporal Grounding: Explicit time markers address unreliable timing inference from relative audio-token positions, especially for long-form audio.At 12.5 Hz, markers inserted every 25 features represent 2-second elapsed-time intervals.

3 Data Pipeline

MOSS-Audio constructs supervision by preserving coherent acoustic events, profiling each segment, and routing it through specialized annotation branches. These outputs are normalized and merged into unified captions for training heterogeneous audio understanding.

  • Segmentation and Profiling: The data engine preserves complete acoustic events and assigns each segment a multi-label audio profile before annotation.This classification-guided strategy produces content-adaptive supervision for heterogeneous real-world audio.
  • Segmentation and Profiling: Segmentation uses detected event boundaries instead of fixed time windows, producing acoustically coherent clips with intact events.Frame-level sound-event detection supplies timestamped AudioSet labels that guide merge-and-cut processing.
  • Segmentation and Profiling: Each segment retains event labels that are aggregated into nine coarse audio categories using duration accounting without double-counting overlaps.The categories include speech, singing, music, natural sounds, environmental background, animals, and other sound types.
  • Speech Branch: Speech segments are transcribed with multiple ASR systems, filtered using cross-model WER consistency, and aligned to the waveform for precise timestamps.Forced alignment produces word-level timestamps that are later aggregated into sentence-level segments.
  • Branch-Specific Annotation: The speech-caption branch adds speaker-aware regions and voice descriptions, while the general-audio branch describes scenes, sources, events, acoustic attributes, and temporal relations.The music branch combines holistic descriptions with symbolic and structural musical evidence before generating listener-facing captions.
  • Caption Merge & Refine: Multiple branch outputs are organized into common slots and merged into a unified natural-language caption for pretraining.The merge interface decouples downstream processing from dataset-specific schemas and combines ASR, speech, event, and music evidence.

4 Pretraining

MOSS-Audio pretraining establishes audio–language alignment through transcription, captioning, and text-only objectives before later instruction and reasoning stages. The recipe balances large-scale data while preserving general language capability and stabilizing the audio pathway.

  • Objectives: Pretraining first establishes robust audio–language alignment needed for later instruction following and audio-grounded reasoning.The report states that later supervised and reinforcement-learning stages depend on an effective audio prefix.
  • Objectives: The objective mixture uses 30% ASR-related tasks, 40% audio captioning, and 30% text-only language modeling across approximately 1.2T training tokens.The three groups jointly cover precise transcription, open-ended audio understanding, and preservation of general text modeling.
  • Objectives: ASR-related pretraining combines ordinary ASR, word-level timestamp ASR, and sentence-level timestamp ASR.These tasks provide lexical transcription together with fine-grained and stable temporal alignment.
  • Objectives: Audio captioning trains the model to generate merged captions assembled from ASR, speech, music, and general-audio annotations.This objective targets unified understanding across heterogeneous annotation branches.
  • Objectives: Text-only data spanning mathematics, code, education, literature, and general text helps preserve the decoder’s original language capability.It is mixed with audio-conditioned data to reduce degradation of general language abilities during audio-language pretraining.
  • Data Mixing: Square-root dataset mixing reduces domination by very large datasets while retaining greater sampling for larger datasets.Each dataset receives probability proportional to the square root of its size rather than its raw size.
  • Training Stages: Pretraining proceeds in two stages, first stabilizing the adapter and DeepStack audio-prefix pathway, then optimizing the full model.Stage 1 uses audio-text objectives without default text-only mixing; Stage 2 expands optimization to the full model.

5 Post-Training

MOSS-Audio uses staged post-training to separate direct instruction following from reasoning-oriented audio understanding. Supervised fine-tuning, reasoning cold start, and DAPO reinforcement learning progressively improve task execution, evidence-grounded analysis, robustness, and format compliance.

  • Post-training overview: Staged post-training produces instruction-following and reasoning-capable variants through supervised fine-tuning, reasoning cold start, and reinforcement learning.SFT adapts the pretrained model to user-facing audio tasks, cold start initializes structured reasoning, and DAPO further optimizes correctness and robustness.
  • Supervised fine-tuning: The SFT mixture covers audio question answering, captioning, ASR, timestamp ASR, and self-identity data, producing instruction-following variants.The stage trains task-specific output formats and consistent responses across different audio questions.
  • Reasoning cold start: Reasoning cold start combines audio-centered and text-only reasoning data to teach evidence-grounded analysis and multi-step reasoning before reinforcement learning.Audio examples connect answers to speech, paralinguistic cues, acoustic events, temporal relations, and music structure.
  • Reinforcement learning: DAPO reinforcement learning samples multiple responses across audio domains and updates the policy using rewards for correctness, response quality, format compliance, and reasoning usefulness.The stage moves beyond fixed teacher reasoning templates through online sampling and reward-based comparison.
  • Reinforcement learning: Dynamic filtering discards zero-variance rollout groups, concentrating updates on prompts with positive and negative trajectories while requiring additional sampling to refill batches.Extra over-sampling rounds rise from 1 to 14 by step 139 as more prompts are solved consistently.
  • Reinforcement learning: Rollout reward rises from approximately 0.69 to above 0.82 while response length stabilizes around 250–270 tokens after early fluctuations.Response length reaches 171.065 tokens at step 12 and 334.324 tokens around step 33 before stabilizing; maximum raw reward is 0.847 at step 136.

6 Evaluation

MOSS-Audio is evaluated across general audio understanding, speech captioning, ASR, and timestamp-aware ASR. The results show complementary strengths: Thinking variants lead broad audio reasoning, while Instruct variants lead speech captioning, transcription, and temporal alignment.

  • Evaluation setup: Evaluation covers general audio understanding, speech captioning, ASR, and timestamp-aware ASR across high-level comprehension and speech-centric perception.The general audio suite includes MMAU, MMAU-Pro, MMAR, and MMSU.
  • General Audio Understanding: 71.08 is the highest open-source average score obtained by MOSS-Audio-8B-Thinking across MMAU, MMAU-Pro, MMAR, and MMSU.MOSS-Audio-4B-Thinking also outperforms several larger 8B-scale open-source baselines.
  • General Audio Understanding: Thinking variants consistently outperform their paired Instruct variants at both 4B and 8B scales on broad audio understanding benchmarks.The reported pattern identifies the reasoning-oriented branch as more suitable for broad audio understanding tasks.
  • Speech Captioning: MOSS-Audio-8B-Instruct achieves the highest speech-captioning average score of 3.7252 across 13 judged dimensions.MOSS-Audio-4B-Instruct follows with 3.7105, with evaluation covering speaker attributes, prosody, voice quality, speaking style, and utterance-level summarization.
  • ASR: MOSS-Audio-8B-Instruct achieves the best overall CER of 11.30 across 12 ASR evaluation dimensions.The evaluated dimensions include dialectal, singing, noisy, multi-speaker, whisper, and far-field speech.
  • Timestamp ASR: MOSS-Audio-8B-Instruct achieves the strongest timestamp ASR performance, with lower AAS indicating more accurate temporal alignment.AAS measures average absolute timestamp shift in milliseconds between predicted and reference timestamp slots.

7 Related Work

Related work spans unified speech-text modeling, broader audio-language models, audio representation learning, and temporal grounding. MOSS-Audio extends multi-level feature integration to audio by routing encoder states into the language model for acoustic and semantic evidence.

  • Unified speech-text modeling: Unified speech-text models established shared sequence modeling for speech and text, supporting spoken-language processing and speech interaction within common architectures.Examples include SpeechT5, Unified Speech-Text Pre-training, SPIRIT-LM, Moshi, Mini-Omni, GLM-4-Voice, Baichuan-Audio, and Step-Audio.
  • Large audio-language models: Large audio-language models extend speech-centric systems to speech, paralinguistic information, non-speech events, music, and broader audio understanding.This line includes LTU-AS, SALMONN, and Qwen-Audio.
  • Audio representation learning: Audio representation learning provides the front-end features that language models use for speech recognition and general audio understanding.The cited work includes HuBERT, WavLM, BEATs, DeepStack, and Qwen3-VL.
  • Audio representation learning: MOSS-Audio adapts DeepStack to audio by routing multi-level encoder states into the language model, preserving low-level acoustic cues and high-level semantic evidence.The design is intended for downstream audio reasoning rather than relying on a single representation level.
  • Temporal grounding: Temporal grounding work targets when events occur, how speaker turns evolve, and which acoustic evidence supports time-sensitive answers.Timestamped transcription is presented as an established practical target for speech recognition.

8 Conclusion

MOSS-Audio combines unified audio understanding, DeepStack feature injection, explicit temporal representation, branched annotation, and staged training in one model family. Its 8B Thinking and Instruct variants lead complementary broad-understanding and speech-oriented evaluations, supporting a unified foundation for heterogeneous audio tasks.

  • Conclusion: MOSS-Audio combines a dedicated audio encoder, DeepStack injection, explicit time-aware representation, branched annotation, and staged training for heterogeneous audio understanding.The model family targets speech, environmental sound, music, captioning, time-aware question answering, and complex reasoning.
  • Conclusion: MOSS-Audio-8B-Thinking leads broad general audio understanding, while MOSS-Audio-8B-Instruct performs best on speech captioning, ASR, and timestamp ASR.The complementary results span open-domain acoustic comprehension and precise speech-oriented tasks.
  • Conclusion: A single model family covers descriptive understanding, transcription, temporal grounding, and reasoning over heterogeneous audio without disconnected specialist systems.The paper positions this coverage as a foundation for future voice agents that interpret user intent, acoustic context, temporal events, and reasoning-relevant cues.

A.1 Evaluation Prompts

The evaluation template presents audio tasks through a system instruction, an audio-containing user turn, and a constrained assistant response. For ASR, the response is restricted to the speech transcription.

  • The shared template consists of a system prompt, a user message containing audio and an instruction, and an assistant response.
  • Automatic Speech Recognition asks the model to transcribe the audio into text.
  • ASR responses must contain only the transcription, without explanations, headings, or additional comments.

Speech Captioning

The evaluation prompts cover audio question answering, multiple-choice selection, human-voice description, and timestamped transcription. Outputs are constrained to task-specific formats, including JSON fields, option letters, and timestamped text.

  • The voice-feature prompt requests a JSON object describing gender, age, pitch, speed, volume, clarity, fluency, accent, texture, emotion, tone, personality, and summary.
  • Sentence-level timestamp ASR serializes each segment as [xx.xx]text[yy.yy], with timestamps rounded to two decimal places and consecutive segments concatenated.
  • The word-level timestamp prompt requests each word in timestamped form and forbids explanations, headings, and line breaks.
  • Open-ended audio question answering asks the model to return an answer based on the audio content.
  • Multiple-choice audio question answering requires selecting one option from A, B, C, or D and returning only its letter.

A.2 Timestamp Serialization Examples

The report illustrates timestamp serialization with word- and sentence-level examples, then evaluates the audio encoder and DeepStack across general understanding, speech recognition, and non-speech captioning.

  • Timestamp serialization examples: The timestamp examples use bracketed start and end times around Chinese words or sentences, with consecutive segments directly concatenated.
  • Evaluation results: ASR results are reported as CER (%), where lower values indicate better recognition accuracy; grouped datasets generally remain in tuple-style rows.
  • Audio encoder capability: The XARES-LLM evaluation compares MOSS Audio Encoder with whisper-large-v3 and Qwen3-Omni-30B-A3B-Instruct across general understanding and generative tasks.
  • Audio encoder capability: Across 15 general-audio benchmarks, the MOSS Audio Encoder significantly outperforms whisper-large-v3 overall and achieves the best results on ASVspoof, ESC-50, and FSD50k.
  • Audio encoder capability: 0.673 is the MOSS Audio Encoder’s overall state-of-the-art score on the generative track covering ASR and audio captioning.
  • In-depth ASR capability: 16.31% versus 17.61% is the average CER or WER across 38 speech datasets for the MOSS Audio Encoder and the comparison encoder, respectively.
  • DeepStack ablation: DeepStack improves overall DATE performance on MECAT-Caption, with gains across music, pure sound, and environmental acoustics despite slight degradation in Pure and Mixed Speech.
  • DeepStack ablation: Intermediate-layer injection restores low-level acoustic, timbral, and environmental cues that the ASR-specialized top layer can overshadow, without adding audio-backbone parameters.
Loading 2606.01802v3…