PaperlayerView exact revision on arXiv

Source-linked AI summary

Fish Audio S2 Technical Report

Shijia Liao, Yuxuan Wang, Songting Liu, Yifan Cheng, Ruoyi Zhang, Tianyu Li, Shidong Li, Yisheng Zheng, Xingwei Liu, Qingzheng Wang, Zhizhuo Zhou, Jiahua Liu, Xin Chen, Dawei Han

arXiv:2603.08823v2cs.SDcs.AIcs.CL

TL;DR

Fish Audio S2 targets scalable, fine-grained controllable TTS, where natural-language vocal annotations and RL alignment remain limited. It combines a Dual-AR model with a dual-purpose data pipeline and multi-reward post-training, achieving strong benchmark performance and production-ready streaming while releasing its weights and tools openly.

  • Problem

    Fine-grained vocal instruction annotation is difficult to scale, and TTS RL alignment can suffer distribution shift when reward models are trained independently from pre-training data.

  • Method

    Fish Audio S2 combines Dual-AR generation, a data pipeline whose quality and ASR models serve as filters and rewards, and GRPO-based multi-reward alignment.

  • Results

    Fish Audio S2 shows strong objective benchmark performance and surpasses previous SOTA by 30% in the instruction-rewritten audio Turing test setting.

  • Takeaways & Limitations

    The released system provides fine-grained natural-language control, coherent long-form synthesis, native multi-speaker multi-turn generation, and public weights, code, and inference tools.

  • Takeaways & Limitations

    The Fish Audio Instruction Benchmark has limited data diversity, imbalanced acoustic-tag distributions, and early-stage human–model alignment analysis.

Abstract

from arXiv · show

We introduce Fish Audio S2, an open-sourced text-to-speech system featuring multi-speaker, multi-turn generation, and, most importantly, instruction-following control via natural-language descriptions. To scale training, we develop a multi-stage training recipe together with a staged data pipeline covering video captioning and speech captioning, voice-quality assessment, and reward modeling. To push the frontier of open-source TTS, we release our model weights, fine-tuning code, and an SGLang-based inference engine. The inference engine is production-ready for streaming, achieving an RTF of 0.195 and a time-to-first-audio below 100 ms.Our code and weights are available on GitHub (https://github.com/fishaudio/fish-speech) and Hugging Face (https://huggingface.co/fishaudio/s2-pro). We highly encourage readers to visit https://fish.audio to try custom voices.

1 Introduction

Fish Audio S2 is an open TTS system designed for controllable, expressive, multilingual speech generation, including long-form, multi-speaker, and multi-turn scenarios. It combines a unified data and RL framework with natural-language instruction following and publicly released deployment resources.

  • The data pipeline combines speech-quality assessment and rich-transcription ASR models for filtering, annotation, and RL reward modeling.The same models are reused as reward signals during alignment.
  • Multi-reward GRPO jointly optimizes semantic accuracy, acoustic quality, and speaker similarity to balance expressiveness and robustness.
  • Natural-language control enables fine-grained instruction following for speech generation, including prosody, emotion, and speaking style.
  • Fish Audio S2 natively generates interleaved multi-speaker, multi-turn dialogues and maintains coherent, stable audio over extended durations.
  • Across public benchmarks, Fish Audio S2 reports consistently strong objective performance, including leading results on the Seed-TTS benchmark and robust multilingual intelligibility and speaker similarity.
  • The authors release model weights, fine-tuning code, and an SGLang-based inference engine through GitHub and Hugging Face.

2 Architecture

Fish Audio S2 uses a Dual-Autoregressive architecture to avoid the sequence-length cost of flattening RVQ codebooks. A large temporal semantic model plans speech content while a lightweight depth-wise decoder reconstructs acoustic detail efficiently.

  • Streaming Architecture: Causal convolutions and sliding-window Transformer blocks support low-latency streaming with bounded memory during long-form inference.
  • Semantic Representation: Semantic distillation makes the first RVQ codebook retain linguistic and phonetic information, supporting planning of linguistic content and coarse prosody.
  • Dual-Autoregressive Generation: Flattening 10 RVQ codebooks along time would increase sequence length tenfold, so Dual-AR separates temporal semantic modeling from depth-wise acoustic modeling.
  • Slow AR: The Slow AR uses a pretrained Qwen3-4B backbone to autoregressively predict semantic tokens interleaved with text and audio tokens.
  • Fast AR: The Fast AR is a lightweight four-layer Transformer that generates the remaining N−1 fine-grained acoustic tokens conditioned on the Slow AR hidden state.
  • Multi-Codebook Fusion: Multi-Codebook Fusion aggregates all N codebook tokens at time t into one continuous embedding for the Slow AR at time t+1.

3 Data Pipeline

Fish Audio S2 addresses scalable TTS data curation by combining quality filtering with rich speech transcription and vocal annotation. The same evaluation engines are reused during RL post-training, aligning data curation and reward modeling.

  • The pipeline addresses manual annotation bottlenecks and distribution shift from independently trained RL reward models.
  • Reward Modeling: The quality and ASR models are reused as RL rewards, penalizing hallucinations, missing words, and ignored vocal instructions.
  • Three-Stage Pipeline: Its three stages separate speech, filter quality, and generate rich transcripts with vocal features and speaker turns.
  • Stage 1: Source Separation and Segmentation: Source separation and VAD isolate clean speech and divide continuous audio into utterance-level segments.
  • Stage 2: Quality Filtering: A speech quality model filters samples using dimensions including signal-to-noise ratio, speaker consistency, recording quality, and intelligibility.
  • Stage 3: Rich Transcription: A rich-transcription ASR model jointly transcribes speech and annotates speaker turns, emotion, prosody, and paralinguistic features as natural-language instructions.These annotations support zero-shot controllable generation.

4 Training

Fish Audio S2 uses a staged training framework combining tokenizer training, progressive language-model alignment, supervised fine-tuning, and RL-based post-training. Its training design supports localized natural-language acoustic control while jointly optimizing semantic accuracy, acoustic quality, and speaker similarity.

  • Training Pipeline: The four-stage pipeline trains an audio tokenizer, performs large-scale pre-training and SFT, and finishes with RL-based post-training.The stages progressively establish discrete audio representations, align the language model, and refine generation quality.
  • Audio Tokenizer: 446M parameters and 1M training steps define the audio tokenizer, which uses multi-period, multi-resolution, and multi-scale STFT discriminators.The discriminators target periodic signals, spectral consistency, and high-frequency detail with phase coherence.
  • Pre-training: Over 10 million hours of raw audio across approximately 80 languages and dialects support two-stage pre-training with context lengths of 8,192 and 16,384 tokens.The second stage enables long-form audio synthesis and multi-turn, multi-speaker conversational generation.
  • Instruction Control: Localized instructions such as whisper, angry, and laugh are inserted at word or phrase positions to control acoustic details without dedicated control tokens.Autoregressive training learns mappings between textual cues and localized acoustic variations.
  • RL-based Post-training: The RL-based post-training phase targets hallucinations, token skipping, and timbre drift while avoiding a value network through a GRPO- and Dr.GRPO-inspired algorithm.Advantages are computed from candidate rewards relative to the intra-group mean, without intra-group standard-deviation normalization.
  • RL-based Post-training: The composite reward combines semantic accuracy, acoustic preference, and timbre similarity signals from ASR, speech-quality, and voiceprint models.The ASR reward also penalizes incorrect speaker IDs and missed vocal instructions.

5 Inference Engine

Fish Audio S2 uses SGLang to serve its Dual-AR speech model with streaming-oriented scheduling, caching, and audio-token interfaces. On a single NVIDIA H200, it achieves low latency, high throughput, and efficient repeated-voice reuse.

  • Serving Architecture: SGLang provides continuous batching, paged KV caching, CUDA graph replay, and RadixAttention for efficient prefix caching.These optimizations maximize GPU utilization and reduce generation latency.
  • Serving Architecture: The Dual-AR architecture fits SGLang’s native autoregressive execution, leaving the core scheduler and execution engine agnostic to audio modality.The paper reports that high performance does not require massive engine modifications.
  • Serving Architecture: The engine adds mixed semantic-acoustic I/O and multi-token RadixCache keys to cache diverse reference-audio contexts.These adaptations support streaming acoustic-token outputs and improve KV-cache hit rates for reference audio.
  • Performance: 0.195 RTF and 100 ms TTFA are achieved on a single NVIDIA H200 GPU in the production serving environment.The reported TTFA is as low as 100 ms.
  • Performance: 3000+ acoustic tokens per second are sustained under high concurrency while keeping RTF below 0.5.The reported operating point supports real-time streaming synthesis under heavy load.
  • Performance: 86.4% average and over 90% peak prefix-cache hit rates make repeated voice requests largely skip reference-audio prefill.This makes prompt-processing overhead nearly negligible for reused voices.

6 Evaluation

Fish Audio S2 is evaluated with objective metrics and LLM-as-a-Judge benchmarks covering intelligibility, multilingual voice cloning, long-form synthesis, naturalness, and instruction following. Across these evaluations, it reports strong multilingual and instruction-sensitive performance, including leading benchmark results and improved instruction-following metrics over Fish Audio S1.

  • Evaluation Setup: Fish Audio S2 is evaluated using objective metrics and LLM-as-a-Judge assessments across intelligibility, speaker similarity, long-form quality, naturalness, and instruction following.The objective track reports WER, CER, and speaker similarity, while the LLM-as-a-Judge track assesses higher-level generation qualities.
  • Objective Evaluation: Fish Audio S2 achieves leading WER on Chinese and English Seed-TTS-Eval splits while remaining competitive on ZH-hard.The result indicates clearer and more stable pronunciation in voice-cloning evaluation.
  • Multilingual Evaluation: Fish Audio S2 achieves the lowest WER in 11 of 24 Minimax languages and the highest SIM in 17 of 24, while reducing CV3-Eval average error from 3.96 to 3.01.The CV3-Eval reduction is a 23.9% relative decrease compared with Fish Audio S1; advantages remain in some low-resource languages for competing systems.
  • Audio Turing Test: Fish Audio S2 surpasses previous state-of-the-art performance in both Audio Turing Test settings, with a 30% advantage in the instruction-rewritten setting.The evaluation also reports that LLM-rewritten instructions improve results over unrewritten baselines.
  • EmergentTTS-Eval: 81.88% overall win rate on EmergentTTS-Eval exceeds the 50% baseline by +31.88 points, with leading win rates in paralinguistics, questions, and syntactic complexity.Fish Audio S2 has overall WER of 8.15, which is not the lowest among the evaluated models, while perceived quality is stronger in instruction-sensitive scenarios.
  • Fish Audio Instruction Benchmark: On the Fish Audio Instruction Benchmark, Fish Audio S2 improves TAR, Naturalness, and Expressiveness over Fish Audio S1 in both Chinese and English.Chinese metrics increase from 0.942/4.15/4.65 to 0.984/4.40/4.94, while English metrics increase from 0.626/3.71/3.93 to 0.881/4.21/4.50.

7 Conclusion

The report presents Fish Audio S2 as a controllable, long-form, multi-speaker and multi-turn TTS system with production-oriented inference. Its design combines architectural, data-pipeline, and reinforcement-learning contributions, supported by objective and LLM-as-a-Judge evaluations.

  • Conclusion: Fish Audio S2 supports fine-grained natural-language control, long-form coherent synthesis, native multi-speaker multi-turn generation, and ultra-low RTF and TTFA in production.The report frames these capabilities as the system’s principal scope.
  • Conclusion: The system combines a Dual-AR architecture, a dual-purpose data pipeline, and RL-based post-training with rewards for semantic accuracy, acoustic quality, and speaker similarity.The data models serve as both pre-training filters and RL reward signals, eliminating distribution shift between stages.
  • Conclusion: Objective benchmarks and LLM-as-a-Judge benchmarks demonstrate effectiveness across intelligibility, speaker similarity, naturalness, and instruction following.The report also introduces the Fish Audio Instruction Benchmark for fine-grained tag following beyond conventional WER and MOS metrics.

8 Author

The listed core contributors are Shijia Liao, Yuxuan Wang, Songting Liu, and Yifan Cheng.

  • Author: The core contributors are listed in order of contribution as Shijia Liao, Yuxuan Wang, Songting Liu, and Yifan Cheng.

A Fish Audio Instruction Benchmark

The Fish Audio Instruction Benchmark evaluates fine-grained instruction following through tag activation, acoustic naturalness, and global expressiveness, using Gemini 3 Pro to score generated speech. Human-model validation shows moderate-to-positive alignment, while also identifying a gap in fine-grained absolute scoring.

  • A Fish Audio Instruction Benchmark: Conventional WER and MOS metrics are described as insufficient for evaluating fine-grained emotional control and paralinguistic expression.The benchmark targets cues such as breaths, laughs, sighs, and emphasis.
  • A.2 Evaluation Method: Gemini 3 Pro evaluates instruction following from both local and global perspectives.The evaluation defines three metrics for the benchmark.
  • A.2 Evaluation Method: Tag Activation Rate measures the percentage of inline tags whose target vocal event is triggered at the exact specified position.Examples of inline tags include [laugh] and [inhale].
  • A.2 Evaluation Method: Acoustic Naturalness uses a 1–5 scale to assess whether triggered vocal events sound realistic and human-like or contain synthetic artifacts.The passage lists robotic timbre, distortion, and abrupt insertions as example artifacts.
  • A.2 Evaluation Method: Global Expressiveness is one of the benchmark’s three defined metrics for assessing fine-grained instruction following.The supplied metric description is truncated after introducing the measure.
  • A.2 Evaluation Method: The evaluation synthesizes speech for each sample, sends the audio with text and inline tags to Gemini 3 Pro, and aggregates scores across languages and scenarios.This procedure produces overall performance measures for the three benchmark dimensions.
  • A.3 Human-Model Alignment Validation: On objective acoustic-event detection, Gemini 3 Pro reaches 76.2% absolute agreement accuracy with human experts and Cohen’s kappa of 0.47.These values indicate moderate agreement for basic event detection.
  • A.3 Human-Model Alignment Validation: For subjective ratings, Naturalness and Expressiveness correlate positively with human judgments, although fine-grained absolute scoring remains imperfect.Pearson correlations are 0.55 and 0.42, with QWK scores of 0.36 and 0.47, respectively.

A.1 Benchmark Design

The benchmark targets fine-grained acoustic-event instruction following by embedding word-level tags in speech transcriptions. It combines English and Chinese dialogue or monologue data, then applies automated processing and expert refinement.

  • Word-level acoustic-event tags are embedded directly into transcriptions to evaluate fine-grained control rather than relying on global emotion prompts.Examples include [laugh], [whispers], [inhale], [exhale], and [emphasis].
  • The benchmark uses utterance segments from multi-turn dialogues and monologues while retaining contextual information around tagged text positions.
  • MELD represents the English setting, while a game character voice dataset represents the Chinese setting.Both subsets contain approximately 500 utterances.
  • Automated data processing is followed by expert verification and fine-grained refinement to remove noise, correct alignment, and place inline tags precisely.

A.4 Conclusion, limitations and future work

The Fish Audio Instruction Benchmark provides a dedicated testbed for fine-grained instruction following, but its first release remains limited in diversity, tag balance, and human–model alignment analysis.

  • The first benchmark release has limited data diversity, imbalanced acoustic-tag distributions, and early-stage human–model alignment analysis.
  • Future versions will expand the dataset and improve automated evaluation to establish a more comprehensive and reliable standard.
Loading 2603.08823v2…