Fish Audio Drama 3 Technical Report

1 Introduction

• Fish Audio S2 lacked support for long directed sessions, revising a take from a later note, local word repair, and preference-based rewards suited to expressive delivery.S2's full-attention backbone handles long references and earlier takes, while its transcription and similarity reward proxies can penalize expressive departures such as laughter, pauses, or whispers.Source: §1 · p. 1
• Drama 3 pairs a hybrid linear-attention model and codebook-MTP decoder with conversational task formats and reinforcement learning using one learned human-preference reward.Decoder layers predict residual codebooks while attending over the sequence, including reference-audio detail; RL uses GRPO without a KL penalty and large batches.Source: §1 · p. 2
• The model supports multi-turn directed takes, multi-character scenes, speech infilling, long-form synthesis, and streaming text input in one system.A later note can request a redo with earlier takes kept in context; scenes can include multilingual lines in the same voice.Source: §1 · pp. 2–3
• On one H200, Drama 3 decodes a single stream at 271 frames per second versus 177 for Fish Audio S2, despite having more than seven times as many parameters.The report attributes serving speed to a decoder that runs once per frame and inference-engine optimizations for its backbone and decoder.Source: §1 · p. 3
• Each Step-Audio-Edit revision raises emotion and style accuracy; Drama 3 has the lowest compared error rates in five languages and lower editing error than Cartesia Sonic-3 in English and Chinese.For Chinese insertion and substitution, it also has higher speaker similarity than dedicated editing models.Source: §1 · p. 3
• The model weights are publicly released on GitHub and Hugging Face, with inference available through SGLang-Omni.The stated goal of the release is to accelerate research and lower the barrier to expressive speech generation.Source: §1 · p. 3

2 Architecture

• Drama 3 replaces S2’s full-attention backbone and per-frame Fast AR with a hybrid linear-attention backbone and a time-causal codebook-MTP decoder.Only two of ten codebooks feed back into the backbone; reference and prompt audio enter as audio tokens.Source: §2 · p. 3

2.1 Audio Tokenizer

• The reused causal tokenizer encodes 44.1 kHz audio into ten-codebook frames at about 21.5 frames per second.Each frame occupies one backbone position, regardless of how many codebooks it contains.Source: §2.1 · p. 3

2.2 Backbone

• The 40-layer Qwen3.5-35B-A3B hybrid backbone combines Gated DeltaNet and full-attention layers, with about 3B of 35B parameters active per token.Its linear-attention layers carry history in constant-size state, while every fourth layer keeps a key–value cache and retains exact whole-context retrieval.Source: §2.2 · p. 3

2.3 Partial Codebook Fusion

• Drama 3 feeds back only semantic code 0 and acoustic code 1, limiting exposure to its sampled codes while retaining a coarse acoustic view.Codes 2–9 bypass the backbone and enter the decoder’s first layer.Source: §2.3 · p. 4

2.4 Codebook-MTP Decoder

• Four shared layers combine backbone state, the frame’s semantic code, and input-frame codes 2–9; nine unshared layers then predict the acoustic codebooks in sequence.At inference, each frame uses one backbone step followed by 13 single-position decoder steps.Source: §2.4 · pp. 5–6
• Training teacher-forces every code in one extra causal pass parallel over time, while inference samples each per-codebook layer before the next runs.The sampled semantic and first acoustic codes feed the next backbone input; codes 2–9 go to the first shared layer.Source: §2.4 · p. 6
• Unlike S2’s Fast AR, each full-width per-codebook layer attends causally over the sequence, directly accessing fine reference codes and prior frames.Reference frames receive the same code injections, but their loss is switched off.Source: §2.4 · p. 6

3 Training

• Drama 3 is initialized from a pretrained language model and trained with three pre-training stages, instruction fine-tuning, and reinforcement learning from human preference.Source: §3 · pp. 6–7

3.1 Pre-training

• Pre-training progresses from text-to-audio and multi-turn TTS to whole recordings, streaming input, and speech infilling, while combining text, semantic-code, and codebook objectives.The recipe uses transcribed speech and text documents; its loss terms are averaged over their own buckets, and the codebook loss back-propagates into the backbone.Source: §3.1 · p. 7

3.2 Instruction SFT

• Instruction fine-tuning teaches natural-language direction, multi-turn take revision, multi-speaker dialogue, and whole-scene dubbing while retaining cloning, long-form, and infilling tasks.Directions include vocal-event and delivery tags, style prefixes, and director’s notes; scene lines are provided up front with supervised separators.Source: §3.2 · pp. 7–8

3.3 Reinforcement Learning from Human Preference

• Drama 3 uses a single learned human-preference reward model instead of S2’s three rewards, which could penalize expressive speech or miss fine-grained control.The reward model reads the policy prompt and take tokens, including references and script, and is trained on paired human preferences; scores are standardized, clipped, and converted to within-prompt win rates.Source: §3.3 · p. 8
• Reinforcement learning uses GRPO with centered advantages, large prompt batches, short schedules, and bounded policy updates; the codebook-MTP decoder remains frozen and receives no RL loss.Each step uses 1,536 prompts, and an asymmetric clip bounds train–inference numerical mismatch; the schedule is designed to follow broad preference and avoid reward over-optimization observed with small-batch, many-step schedules.Source: §3.3 · pp. 8–9

4.1 Generation Structure

• Drama 3 reads about 6.0 GB of weights per frame, less than two fifths of S2’s 15.6 GB despite having more than seven times as many parameters.Its decoder reads each of 13 layers once per frame, while block-scaled FP8 experts use one byte per parameter instead of two.Source: §4.1 · p. 9, Table 1 · p. 10
• 271 frames per second on one H200 versus 177 for S2; Drama 3 also reached 295 frames per second on one B300.At concurrency 32 on H200, FP8 experts raised the per-stream decode rate by 36% over BF16 experts.Source: §4.1 · pp. 9–10

4.2 Kernel Launches Below CUDA Graphs

• Kernel-launch overhead motivated removing auxiliary operations and fusing work, while retaining necessary MoE launches and rejecting a slower fused router kernel.Decode changes removed 100 kernels per step and raised per-stream rate 4.2% at concurrency 1; prefill graph capture cut host time per round from 30.9 to 14.7 ms and raised rate 15.3% at concurrency 32 on an earlier development model.Source: §4.2 · pp. 10–11, Table 2 · p. 11

4.3 Shape-Driven Kernel Optimization

• Kernel tuning follows serving shapes through capture, profiling, candidate testing, in-engine validation, and serving A/B comparison before deployment.Nine cases showed that kernel and serving verdicts could disagree; after issues were resolved, the Gated DeltaNet prefill kernel raised throughput 11.1% at concurrency 256. The FP8 MoE kernel cut MoE layer time at M = 256 by 19.2%.Source: §4.3 · p. 12

4.4 Prefill–Decode Disaggregation

• Prefill–decode disaggregation moves the two phases to separate GPUs and transfers each completed prefill’s KV entries and recurrent state to decode.On one GPU, every prefill pauses running streams; separating phases prevents those prefill rounds from interrupting playback.Source: §4.4 · pp. 12–13
• 24 streams per GPU met the no-stall, first-percentile 35-fps bar in a capacity test, versus about 10 streams per GPU on a single GPU.In a production canary, PD decode GPUs carried about twice as many streams as single-GPU workers; their first-percentile rates were 55 fps versus 45–47 fps, with equivalent CER within 0.02 points.Source: §4.4 · p. 13

5.1 Evaluation Setup

• Evaluation uses each benchmark’s references and target text, fixed sampling settings, and signal checks that trigger regeneration of failed outputs.The study evaluates a single model across multilingual cloning, editing, and next-turn revision; most baselines are designed for one task.Source: §5 · p. 13, §5.1 · p. 14

5.2 Preference Reward vs. Rule-Based Rewards

• Styled readings lower SIM by 0.019 to 0.310, so similarity reward prefers default readings in 62.0% to 99.7% of pairs, depending on style.Laughing raises WER by 0.79 points, so a WER reward penalizes laughing; Drama 3 instead uses one learned human-preference reward without transcription or speaker-similarity rewards.Source: §5.2 · p. 14
• With direction, the reward model prefers whispered and laughing readings in 62.2% and 64.4% of pairs, versus 10.6% and 27.1% without direction.In every tested style, adding direction raises the share of pairs in which the reward model prefers the styled reading.Source: §5.2 · p. 14, Table 3 · p. 14
• RL lowers flagged first takes from 2.82% to 1.24% on test-zh and from 3.40% to 1.84% on test-en; after regeneration, CER/WER remain lower than SFT.Final-take error rates are 0.83 against 0.84 CER on test-zh and 1.54 against 1.60 WER on test-en.Source: §5.2 · pp. 14–15, Table 4 · p. 15

5.3.1 Cloning from Multiple References

• With three references instead of one, SIM to the benchmark reference rises from 0.605 to 0.633.With one reference, UTMOS is 4.31, above OmniVoice and CosyVoice 3 at 4.28.Source: §5.3.1 · p. 15, Table 5 · p. 15

5.3.2 Multilingual Cloning

• Drama 3 has lower error rates than OmniVoice in 15 languages and than FireRedTTS3-Base in 17; it is below both and Fish Audio S2 in 11 languages.It has lower error rates than every other table system in Chinese, Dutch, English, German, and Turkish; baseline results mix scores quoted from papers with systems run and scored by the authors.Source: §5.3.2 · p. 15, Table 6 · p. 16

5.4 Streaming Synthesis

• Streaming yields WER 1.86 versus 1.83 for non-streamed synthesis, with SIM 0.640 for both modes.The comparison uses all 1,088 Seed-TTS-Eval test-en sentences.Source: §5.4 · p. 16
• In streamed mode, audio is sent out chunk by chunk while the rest of the sentence is still being generated.Non-streamed mode returns the whole take after it is finished.Source: §5.4 · p. 16
• The reported streaming comparison covers 1,088 test-en sentences synthesized individually in streamed and non-streamed modes.Source: §5.4 · p. 16

5.5 Editing by Infilling

• The edit window spans the first through last differing words; Drama 3 regenerates it using the full target text and source audio tokens on either side.Ming-Freeform covers deletions, insertions, and substitutions in English and Chinese; its full split has more varied instructions and substitutions than the basic split.Source: §5.5 · p. 16
• In Chinese, Drama 3 has SIM 0.909 and 0.898 on insertion and substitution, respectively, versus 0.89 for CosyEdit2.It also has the lowest deletion WER, 7.82 against 8.23, and the smallest MAED on deletion and substitution.Source: §5.5 · pp. 16–17, Table 7 · p. 17
• Against Cartesia Sonic-3, Drama 3 has lower WER on both splits in both languages; Chinese WER is 3.69 against 8.64 on basic and 2.35 against 6.53 on full.The comparison pastes each system’s edited window into the source recording.Source: §5.5 · p. 17, Table 8 · p. 17

• Step-Audio-Edit tests voice cloning followed by up to three target emotion or style revisions, or one positioned paralinguistic event; each note revises the latest take.Drama 3 keeps the original reference and most recent take in context; event insertion uses a point tag and infilling.Source: §5.6 · p. 17
• Emotion accuracy rises from 55.5 to 58.2 and style accuracy from 76.5 to 81.8 over three revision rounds; paralinguistic score rises from 1.32 to 2.32.Source: Table 9 · p. 18, §5.6 · p. 18
• Table 9 compares Step-Audio-EditX in its Prompt-Fixed setting with Drama 3 judged using the official Gemini-2.5-Pro protocol.Emotion and style are reported as accuracies (%).Source: §5.6 · p. 17, Table 9 · p. 18

• 271 frames per second on one H200 versus 177 for Fish Audio S2, with Drama 3 having more than seven times as many parameters.Source: §6 · p. 18
• One model handles multi-reference voice cloning, natural-language direction and revision, multi-character scenes, infilling edits, and long-form and streaming audio.Source: §6 · p. 18
• A hybrid linear-attention mixture-of-experts backbone pairs with a codebook-MTP decoder whose layers attend across the sequence to predict acoustic detail.Source: §6 · p. 18
• Reinforcement learning uses a single learned human-preference reward and no rule-based rewards, which would penalize laughs, pauses, and whispers.Source: §6 · p. 18
• The inference engine runs codebook-MTP once per frame, optimizes expert kernels and separates prefill from decode to keep new requests from stalling running streams.Source: §6 · p. 18

• Training loss falls on assistant audio frames, line-separated output separators, and the closing token; infilling also supervises the middle or its seams.Source: A Pre-training Sequence Formats · p. 26
• Stages 1–2 teach plain synthesis and cloning through consecutive recording utterances arranged as user and assistant turns, with references in the system turn.Source: A Pre-training Sequence Formats · p. 26, Figure 7 · p. 26
• Long-form synthesis voices a transcript in one turn, using reference clips cut later in the same recording so context audio does not overlap the target.Source: A Pre-training Sequence Formats · p. 26, Figure 8 · p. 26
• Line-separated synthesis emits one audio segment per text line, with supervised separators anchoring text to audio and limiting how skipped or repeated lines affect later segments.Source: A Pre-training Sequence Formats · p. 26, Figure 9 · p. 27
• Streaming input arrives in consecutive transcript chunks; each user turn includes the current chunk and the following chunk, while an empty following chunk marks the end.Source: A Pre-training Sequence Formats · p. 27, Figure 10 · p. 27

• The prompt formats supervise assistant turns across ten codebooks per frame and the closing token; infilling additionally weights loss toward edit seams.At inference, generation begins after the final voice token and continues until the end token.Source: B Prompt Examples · p. 27
• Two voices are cast from three reference clips, with two clips sharing one speaker token and the third introducing the other voice.Source: B Prompt Examples · pp. 27–28, Figure 11 · p. 28
• Directing examples progress from an absolute style prefix to relative notes, which are interpreted against the audio generated in earlier turns.Source: Figure 12 · p. 28, B Prompt Examples · p. 28
• Scene dubbing supplies all lines up front and casts characters from text descriptions; supervised separators teach one audio segment per line.Source: B Prompt Examples · p. 28, Figure 13 · p. 29
• Infilling replaces a word using the full revised transcript and the original audio before and after it; an optional duration hint specifies the new span length.The suffix precedes the prefix in the prompt so the prefix sits next to the generated audio.Source: B Prompt Examples · pp. 28–29, Figure 14 · p. 29

C Control Syntax Reference

• Structural tokens use the form <|name|> and are single vocabulary entries; every other control is ordinary text.Source: C Control Syntax Reference · p. 29
• Table 10 lists Drama 3’s control syntax, and Section 3 and Appendix B identify where formats are introduced and show complete prompts.Source: C Control Syntax Reference · p. 29, Table 10 · p. 30
• Direction controls are free natural language and include point tags, scoped tags, style prefixes, director’s notes, and relative notes.Point tags act at their position, scoped tags on enclosed spans, and the other listed controls on the whole line.Source: C Control Syntax Reference · p. 29
• Round-parenthesized text is spoken, while language and duration hints condition output without being spoken.The optional language hint names the request’s main language; each line is spoken in the language of its own text.Source: C Control Syntax Reference · p. 29