PaperlayerView exact revision on arXiv

Source-linked AI summary

FireRedAudio: A General-Purpose Audio Language Model with Decoupled Continuous Representations for Understanding and Generation

Junjie Li, Xuelong Geng, Kun Xie, Feiyu Shen, Yichen Wu, Ziqi Dai, Yichen Han, Yan Jia, Kai Huang, Junjie Chen, Yixuan Li, Manzhen Wei, Fenglong Xie, Lei Xie, Xu Tang, Yao Hu

arXiv:2608.24168v1cs.CLcs.SD

TL;DR

FireRedAudio addresses the challenge of using audio representations that are compact for understanding yet reconstructible for speech generation. It decouples continuous input pathways within a shared 9B-parameter LLM, achieving competitive or leading performance across evaluated tasks while supporting long-form understanding and speech generation.

  • Problem

    Audio understanding favors compact representations for long-context modeling, whereas speech generation requires waveform-reconstructible features that preserve fine-grained acoustic detail.

  • Method

    FireRedAudio routes analyzed audio through an Audio Encoder and speech-generation inputs through a RedAE-based pathway, while a shared LLM generates text or conditions a flow-matching DiT for continuous acoustic latents.

  • Results

    FireRedAudio achieves competitive or leading performance across evaluated tasks, supports audio understanding of recordings up to one hour, and provides second-level timestamp accuracy for structured long-form organization.

  • Takeaways & Limitations

    The results support decoupled continuous input representations as a viable basis for unifying audio understanding with continuous-latent speech generation in a moderate-scale model.

Abstract

from arXiv · show

A unified audio model must recognize and understand linguistic, paralinguistic, and environmental information while supporting speech synthesis and editing. A key challenge is representation: understanding favors compact features suited to long-context modeling, whereas speech generation requires reconstructible features that preserve fine-grained acoustic detail. We introduce FireRedAudio, a general-purpose audio language model with a shared 9B-parameter LLM. To the best of our knowledge, it is the first publicly disclosed unified audio-language model to provide separate continuous input representations for understanding and generation within a single trainable autoregressive LLM. Audio to be recognized or analyzed is processed by a dedicated Audio Encoder, while speech inputs for generation use a RedAE-based pathway. The LLM directly generates text or conditions a flow-matching DiT to produce continuous acoustic latents. Through progressive multitask training, FireRedAudio supports ASR and audio understanding, with the latter extending to recordings of up to one hour, as well as zero-shot TTS, Instruct TTS, and semantic and acoustic speech editing. Its structured organization of long-form audio achieves second-level timestamp accuracy. Across comprehensive evaluations, FireRedAudio achieves competitive or leading performance in audio understanding and multilingual ASR, strong content accuracy and speaker preservation in zero-shot TTS, leading instruction following in Instruct TTS, and substantial improvements over Ming-UniAudio-Edit in both semantic and acoustic speech editing. These results demonstrate the viability of decoupled continuous input representations for unifying audio understanding and continuous-latent speech generation in a model of moderate scale. Our code is available at https://github.com/FireRedTeam/FireRedAudio.

1 Introduction

FireRedAudio addresses the mismatch between compact representations for audio understanding and reconstructible representations for speech generation by decoupling continuous input pathways within one shared LLM. It unifies understanding, synthesis, editing, and long-form audio organization, achieving competitive or leading results across evaluated tasks.

  • Core design: The Audio Encoder produces continuous perceptual representations for analyzed audio, while a RedAE-based pathway supplies reconstructible speech representations for generation.The LLM directly generates text for understanding tasks or conditions a flow-matching DiT that generates continuous RedAE latents for waveform rendering.
  • Core design: FireRedAudio uses separate continuous input pathways for understanding and speech generation within one trainable autoregressive LLM.The Audio Encoder serves understanding, while the RedAE-Patch pathway serves speech generation without fusing their input representations.
  • Capabilities: FireRedAudio supports ASR, audio understanding, zero-shot TTS, Instruct TTS, and semantic and acoustic speech editing through a unified model.Its evaluations cover ASR, MMAU and MMSU, Seed-TTS-Eval, InstructTTSEval, and Ming-Freeform-Audio-Edit.
  • Evaluation: With a 9B-parameter LLM backbone, FireRedAudio achieves competitive or leading results across ASR, audio understanding, speech synthesis, and editing evaluations.The contribution passage reports competitive or leading results across the evaluated tasks without assigning a single aggregate score.
  • Long-form audio: FireRedAudio organizes recordings of up to one hour into timestamped entries with second-level timestamp accuracy.The system demonstrates stable time–content alignment under a one-second boundary tolerance on recordings from 5 to 50 minutes.

2 Method

FireRedAudio uses separate continuous audio pathways for understanding and speech generation while sharing one language model. The understanding pathway produces text, whereas the generation pathway conditions a DiT that produces reconstructible RedAE latents for waveform synthesis.

  • Model Overview: FireRedAudio routes each audio segment through an understanding or generation pathway while retaining a shared 9B-parameter LLM.The pathways remain separate rather than fusing their input representations.
  • Model Overview: Understanding inputs pass through an Audio Encoder and Adapter, while speech-generation inputs use RedAE and Patch Encoder representations.The understanding pathway operates at 12.5 Hz, while RedAE latents are grouped into 6.25 Hz RedAE-Patch representations at the LLM interface.
  • RedAE and Patch Encoder: RedAE is a deterministic autoencoder that maps waveforms to continuous latents and reconstructs waveforms without variational posterior modeling or KL regularization.Its hybrid objective combines reconstruction, adversarial, feature-matching, and semantic-distillation losses, with semantic supervision from a frozen teacher Audio Encoder.
  • RedAE and Patch Encoder: 50 Hz audio frames are compressed into 25 Hz 64-dimensional RedAE latents, then four consecutive frames are aggregated into each 6.25 Hz RedAE-Patch representation.The Patch Encoder reduces LLM-side sequence length while preserving the latent pathway used for speech generation.
  • Continuous-Latent Speech Generation: The shared LLM generates text for understanding tasks or supplies stepwise conditioning to a DiT for zero-shot TTS, Instruct TTS, and speech editing.The DiT generates four RedAE latent frames per audio step, which are patched and fed back as context for the next step.
  • Continuous-Latent Speech Generation: Flow-matching training lets the DiT generate continuous RedAE latents from Gaussian noise while the LLM captures instructions and causal sequence context.A local DiT window uses the current and two preceding LLM hidden states, while acoustic conditioning promotes local continuity.

3 Training Strategy

FireRedAudio uses pretrained modules alongside newly initialized interfaces and components. Progressive optimization is motivated by the instability that joint training from the outset would expose to the shared LLM.

  • Training Strategy: The Audio Encoder, LLM, and RedAE are pretrained, while the Audio Adapter, Patch Encoder, and DiT establish newly initialized interfaces.These components connect understanding and continuous-latent speech generation to the shared LLM.
  • Training Strategy: Jointly optimizing all trainable components from the outset would expose the LLM to unstable understanding and speech-generation interfaces.The training strategy therefore avoids beginning with simultaneous optimization of every trainable component.

3.1 Data Mixture and Sampling

FireRedAudio uses hierarchical sampling to prevent abundant tasks and datasets from dominating unified multitask training. Training volume and sequence-length limits are tracked across a five-stage strategy.

  • Raw-scale sampling would let abundant tasks and datasets dominate optimization because training data are imbalanced across task families, languages, domains, and sources.
  • The sampler first selects a task family and then one of its constituent datasets.
  • Task-level and dataset-level smoothing use αtask = 0.5 and αdata = 0.7, with stronger balancing across families than within-family datasets.
  • Task-family weights change across training stages to control how frequently each task is sampled according to stage objectives.
  • Training stages are characterized by data focus, billions of non-padding multimodal tokens, and maximum effective per-example sequence length.

3.2 Progressive Training

FireRedAudio is trained through a progressive curriculum that first aligns the understanding interface, then jointly expands multimodal capabilities and instruction following. A final long-context stage extends audio understanding to one-hour recordings while retaining prior supervision to mitigate forgetting.

  • The first stage uses 80B Chinese- and English-language ASR tokens and 100B multilingual ASR tokens to align the randomly initialized Audio Adapter with pretrained audio and language components.
  • The second stage adds 150B audio-understanding tokens alongside 120B Chinese- and English-language ASR tokens and 120B multilingual ASR tokens to broaden supervision beyond transcription.
  • Mid-training jointly optimizes the Audio Encoder, Audio Adapter, LLM, Patch Encoder, and DiT over 990B multimodal tokens spanning ASR, audio understanding, text, zero-shot TTS, and audio–text interleaving.
  • Post-training introduces 22B Instruct TTS tokens and 12B speech-editing tokens within an approximately 511B-token mixture while retaining earlier task supervision.
  • The final stage expands maximum effective sequence length from 8k to 200k and maximum audio-understanding duration to one hour, adding 80B long-form understanding tokens for approximately 591B total tokens.

3.3 Optimization Configuration

Optimization configuration varies across five stages through module freezing, learning-rate schedules, and packing budgets. These settings accommodate heterogeneous multimodal batches while controlling memory use.

  • Table 2 specifies stage-wise learning rates, schedules, frozen modules, and per-replica packing budgets for the LLM backbone, audio components, Patch Encoder, and DiT.
  • Packed microbatches obey stage-specific multimodal-token and audio-duration limits for understanding and speech generation, keeping per-replica memory use stable.

4 Experiments

FireRedAudio is evaluated across audio understanding, ASR, zero-shot and Instruct TTS, speech editing, and long-form temporal grounding. It achieves strong results across these tasks, including leading or competitive benchmark performance and second-level timestamp accuracy.

  • ASR and Audio Understanding: MMAU and MMSU assess broad audio understanding and fine-grained spoken-language perception and reasoning.
  • ASR and Audio Understanding: FireRedAudio achieves the highest accuracy in all three MMAU and MMSU settings.
  • ASR and Audio Understanding: 14.94% is FireRedAudio’s FLEURS-102 macro-average ASR error rate, the lowest reported among compared systems.
  • Zero-Shot TTS: 1.20% is FireRedAudio’s lowest average content-error score on Seed-TTS-Eval, with average speaker similarity of 0.71.
  • Instruct TTS: FireRedAudio achieves the highest accuracy in all six Chinese and English InstructTTSEval settings.The largest margin is 6.1 percentage points on EN RP.
  • Speech Editing: FireRedAudio matches or improves upon Ming-UniAudio-Edit on all but one reported speech-editing value.The sole exception is Chinese SIM under open deletion, differing by 0.01.
  • Structured Long-Form Audio Organization: FireRedAudio outperforms Qwen3.5-Omni-Plus on all three overall temporal-grounding metrics for recordings lasting 5 to 50 minutes.Its overall pass rate exceeds 96% with a 0.5-second boundary tolerance and remains above 93% in every duration group.

5 Conclusion

FireRedAudio unifies audio understanding and speech generation through decoupled continuous input representations within a shared 9B-parameter LLM. The evaluated results support this design as a viable way to combine broad audio capabilities while retaining second-level temporal grounding.

  • FireRedAudio unifies ASR, audio understanding, zero-shot TTS, Instruct TTS, and semantic and acoustic speech editing within a shared 9B-parameter LLM.
  • A dedicated Audio Encoder serves understanding, while the reconstructible RedAE pathway serves speech generation.The model combines these pathways with a flow-matching DiT and progressive multitask training.
  • FireRedAudio supports audio understanding for recordings up to one hour and achieves competitive or leading performance across evaluated tasks.
  • Structured long-form audio organization achieves second-level timestamp accuracy.
  • The results support input-side representation decoupling as a viable approach for sharing language and reasoning across understanding and speech generation.

Appendix A Detailed Multilingual ASR Results

The appendix details how FLEURS-102 multilingual ASR results are scored across writing systems. It reports language-specific error rates using scoring units matched to each script.

  • FLEURS-102 reports WER for most languages, CER for Chinese, Japanese, and Cantonese, and space-stripped CER for Korean.
  • GCER is used for Lao, Burmese, Thai, and Khmer, treating each extended grapheme cluster as one scoring unit.
  • Table 10 provides per-language ASR error rates for FireRedAudio, Qwen3.5-Omni-Plus, and Gemini 3.1 Pro, with lower values indicating better performance.
Loading 2608.24168v1…