Source-linked AI summary
AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing
Ziyang Ma, Zhikang Niu, Wenming Tu, Tianrui Wang, Ruiqi Yan, Junxi Liu, Yanru Huo, Nickk Huang, Yang Liu, Qicong Xie, Zeyu Xie, Hui Wang, Haitao Li, Zixuan Jiang, Yalin Li, Jie Fang, Yifan Duan, Zeyue Tian, Guangzheng Li, Haina Zhu, Shuyi Wang, Jinwen Wang, Mingyu Cui, Tian Tan, Auden, Sen Liang, Steve Yves, Shan Yang, Liefeng Bo, Zilong Zheng, Kai Yu, Eng-Siong Chng, Xie Chen
TL;DR
AuK addresses the challenge of unifying speech generation and diverse editing tasks that otherwise require fragmented systems and heterogeneous supervision. It uses a shared instruction-and-audio interface with multimodal semantic conditioning, a joint audio VAE, and a hybrid flow Transformer. The model achieves leading performance in speech generation and general instruction-guided editing, remains competitive for restoration, and still depends on task routing and prompt enhancement for unconstrained requests.
Problem
Speech generation, editing, enhancement, and separation have differing constraints, conditioning needs, and evaluation signals, making a single broadly capable and efficiently trained model difficult to realize.
Method
AuK unifies five task families through natural-language instructions and optional audio context, combining an MLLM, shared audio VAE, hybrid flow Transformer, and staged generation–editing alignment.
Results
AuK achieves leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing while remaining competitive on signal-level restoration tasks.
Takeaways & Limitations
AuK provides an open-source common interface for broad speech generation and editing, with useful transfer across utterances, tasks, and languages reported qualitatively.
Takeaways & Limitations
Robust native understanding of unconstrained editing requests remains incomplete, so the system still benefits from explicit task routing and prompt enhancement.
Abstract
from arXiv · showhide
We introduce AuK, an open-source foundational model that unifies speech generation and editing through a common interface of natural-language instructions and audio context. To support this broad capability set, we construct approximately 3.03 billion instruction--audio instances and 1.95 million hours of effective supervision across five task families: speech generation, content editing, enhancement and separation, paralinguistic editing, and acoustic editing. AuK combines a multimodal large language model for semantic conditioning, an VAE jointly trained on speech, general audio, and music for acoustic conditioning, and a hybrid rectified-flow Transformer that performs dual-stream MMDiT blocks followed by unified single-stream DiT blocks for generation. Training begins with generation-only warm-up and proceeds to joint generation--editing pre-training. We then apply complementary post-training strategies: human-feedback preference optimization for open-ended editing and reward-based reinforcement learning for speech generation. To reduce inference cost, we further distill the model with consistency initialization and task-routed Decoupled DMD. The resulting AuK-Flash performs 4-step inference without classifier-free guidance and achieves a 4.5 wall-clock speedup over the full model under matched conditions. Experiments demonstrate leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing, while remaining competitive on signal-level restoration tasks. We release both the source code and model weights to support reproducibility and further research.
1 Introduction
AuK addresses fragmented speech-generation and editing workflows with a unified instruction-and-audio interface. It combines multimodal semantic conditioning, shared acoustic representations, and hybrid flow-transformer generation, achieving leading results across generation and general editing while remaining competitive for restoration.
- Separate task-specific models fragment user experiences across speech synthesis, editing, enhancement, and separation.Users may request style-controlled synthesis, localized replacement, emotion or accent changes, nonverbal insertion, speaker isolation, or restoration.
- Unification is difficult because tasks impose different output constraints, conditioning inputs, and supervision and evaluation requirements.Generation, localized editing, preservation-oriented transformations, and restoration require different combinations of text, audio, and objective or subjective evaluation signals.
- AuK maps natural-language instructions and optional audio context to target waveforms through an MLLM, shared audio VAE, and hybrid flow Transformer.Dual-stream MMDiT blocks exchange semantic and acoustic information before single-stream DiT blocks predict target latents, supporting both text-only generation and reference-conditioned editing.
- AuK trains generation and editing jointly after a generation-only warm-up, then applies human-feedback preference optimization for editing and reward-based reinforcement learning for generation.These post-training stages address open-ended editing and automatic generation rewards with complementary strategies.
- AuK achieves leading performance on speech generation and general instruction-guided editing while remaining competitive on signal-level restoration tasks.The released AuK-Flash retains broad generation and editing capability at substantially reduced inference cost.
2 Data Construction
AuK organizes approximately 3.03 billion instruction–audio instances and 1.95 million hours of supervision into five task families sharing one instruction-conditioned waveform interface. Its data construction combines curated generation data with synthetic, aligned, and transformation-based supervision for acoustic, paralinguistic, and content editing.
- Approximately 3.03 billion instruction–audio instances provide 1.95 million hours of effective supervision across five task families.The families are speech generation, acoustic editing, paralinguistic editing, content editing, and enhancement and separation.
- Speech Generation: Speech generation supervision covers transcript-free zero-shot TTS and instruction-controlled TTS.Zero-shot TTS uses reference speech and target text without providing the reference transcript; Instruct TTS combines target text with natural-language descriptions and structured vocal attributes.
- Speech Generation: Zero-shot TTS creates n × (n −1) bidirectional training instances from n utterances and supports inference from complete or cropped reference speech without transcription.Each pair alternates acoustic prompt and synthesis target, while only prompt speech and target text are supplied.
- Acoustic Editing: Acoustic editing uses deterministic transformations to vary speaking rate, loudness, and pitch while preserving linguistic content and non-target characteristics.Targets include five rate multipliers, six loudness offsets, and six pitch shifts, with validation and peak protection applied.
- Paralinguistic Editing: Paralinguistic editing constructs paired supervision for emotion, timbre, accent, nonverbal vocalization, and whisper-style conversion.The data preserves the utterance’s linguistic content while modifying how it is delivered, using synthesized or aligned references and localized reconstruction.
- Content Editing: Content-editing data supports insertion, deletion, and replacement of speech or lyrics through localized masking and synthesis.Word-level alignment identifies edit intervals, while target construction aims to preserve speaker identity, prosody, melody, and surrounding acoustic context.
3 Model Design
AuK separates semantic and acoustic conditioning before fusing them in a hybrid Transformer that predicts target audio latents. The design supports text-only generation and reference-conditioned tasks through frozen multimodal and VAE encoders with learned semantic aggregation.
- AuK combines an MLLM semantic encoder, an audio VAE, and a FLUX-style Transformer to predict target acoustic latents.The MLLM processes instruction and audio context, the VAE preserves fine-grained acoustic information, and the Transformer fuses both conditions.
- Reference-audio tasks encode the input through both the MLLM audio encoder and VAE, whereas text-only tasks use only instruction input and noisy target latents.After dual-stream processing, both configurations use single-stream blocks and VAE decoding to produce audio.
- MLLM Semantic Condition: Layer-wise MLLM hidden states are combined with learnable weights and layer normalization to form the semantic condition.This aggregation retains complementary linguistic, acoustic, and cross-modal information from different representation depths.
- MLLM Semantic Condition: Audio-conditioned tasks jointly encode the instruction and reference waveform, while text-only tasks derive semantic representations solely from the instruction.The distinction determines whether the MLLM receives an audio representation alongside tokenized text.
- VAE Acoustic Condition: The VAE provides a shared latent space for reference conditioning and waveform reconstruction, with its decoder converting target latents back into audio.Its encoder maps 24 kHz waveforms to 64-dimensional latents at 50 Hz, while the causal decoder reconstructs audio.
- Transformer Backbone: Dual-stream MMDiT blocks enable bidirectional semantic–acoustic interaction while preserving stream-specific residual pathways before single-stream DiT fusion.The fused sequence is processed by subsequent DiT blocks that predict the flow velocity at target-latent positions.
4 Model Training
AuK training combines a shared VAE and Transformer-based flow model with a two-stage curriculum that progresses from speech generation to joint generation and editing. Post-training then uses human feedback for editing and automatic rewards for generation.
- VAE Training: The AuK-VAE learns a compact latent space for reference-audio conditioning and high-fidelity waveform reconstruction.It is trained on speech, music, and general audio using reconstruction, adversarial, feature-matching, and KL objectives.
- VAE Training: The VAE encoder uses six downsampling blocks, while its decoder uses six transposed-convolution blocks based on BigVGAN.The encoder operates on 24 kHz mono waveforms and reduces temporal resolution through strides (2, 2, 2, 3, 4, 5).
- Unified Pre-Training: 50k updates of generation-only warm-up precede 600k updates of joint generation-and-editing training with a shared rectified-flow objective.The MLLM and audio VAE remain frozen while the Transformer backbone and layer-fusion parameters are updated.
- Unified Pre-Training: The flow model predicts target velocity from noisy latents, flow time, and semantic or acoustic conditions, then integrates the learned field with a deterministic ODE solver.Training uses masked mean-squared error and samples flow times to emphasize noisier states while retaining the full trajectory.
- Post-Training: Post-training separates editing preference optimization from generation reinforcement learning to address their different supervision requirements.Editing uses human feedback for subjective task completion and perceptual quality, while generation uses automatic rewards for content, speaker similarity, and style.
5 Model Acceleration
AuK acceleration distills the full iterative, guidance-dependent model into a four-step student. Consistency initialization provides the starting point, while task-routed Decoupled DMD preserves distribution matching and separation capability.
- Distillation Overview: The full model is distilled into a four-step student without classifier-free guidance to reduce inference cost.The distillation pipeline uses consistency initialization followed by task-routed Decoupled DMD.
- Consistency Initialization: Consistency initialization trains the student to predict the clean endpoint directly from noisy states along the teacher’s sampling trajectory.Neighboring student predictions are matched after teacher-guided trajectory advancement.
- Decoupled DMD: Decoupled DMD separates guidance transfer from distribution matching using independently re-noised student predictions.CFG Augmentation transfers conditional guidance, while Distribution Matching aligns the student and teacher distributions.
- Decoupled DMD: Adaptive projected guidance replaces CFG in the guidance-transfer branch to suppress oversaturation, overshoot, and audible clipping.The teacher remains frozen as the real-score model while the consistency checkpoint initializes the student and fake-score model.
- Task-Routed Decoupled DMD: Separation examples bypass both DMD branches and instead use supervised clean-latent regression to preserve separation ability.This routing addresses regressions toward the unprocessed mixture observed when Decoupled DMD is applied uniformly across tasks.
6 Model Inference
AuK inference uses a Prompt Enhancer to convert free-form requests and audio into validated model conditions before flow sampling. The same pipeline supports both the full model and the no-guidance AuK-Flash configuration.
- Prompt Enhancer: The Prompt Enhancer identifies the task, rewrites the request, prepares optional audio, and estimates output duration before AuK sampling.Canonical instructions may bypass the Prompt Enhancer and enter the model directly.
- Prompt Enhancer: Audio-derived ASR and language context help resolve implicit references, distinguish similar operations, and extract task parameters.The extracted parameters are validated against training-supported ranges before instruction rendering.
- Audio Preparation: Input audio receives task-dependent silence handling, resampling, and level normalization before being encoded as an acoustic reference.These operations standardize the input while preserving content that should remain unchanged.
- Duration Estimation: The target duration is estimated from input duration, text byte lengths, or speed-editing parameters and determines the target latent length.Latent length is set at the VAE rate of 50 Hz.
- Inference Configuration: AuK-Flash uses the task-routed distilled checkpoint without guidance, whereas the full model uses its EMA checkpoint with classifier-free guidance.The reported speed comparison uses four-step, CFG-free AuK-Flash against 32-NFE AuK under matched hardware, duration, and batch size.
7 Performance
AuK delivers strong results across speech generation, general editing, and signal-level restoration, while AuK-Flash preserves broad capability with different quality–efficiency trade-offs. The full model generally favors linguistic accuracy and edit fidelity, whereas Flash often favors perceptual quality and speaker preservation.
- Enhancement and separation: AuK-Flash often provides stronger perceptual quality, while AuK more often preserves linguistic content, edit fidelity, or speaker identity across restoration tasks.Flash leads UTMOS on enhancement, perceptual scores on separation and super-resolution, and speaker similarity on acoustic editing; the full model generally has lower recognition error.
- Reconstruction: AuK-VAE achieves the best reconstruction result on all four metrics across speech, general audio, and music.The results indicate consistent preservation of perceptual quality, intelligibility, and spectral detail across all three domains.
- Speech generation: 2.65% average recognition error and 0.795 average SIM make AuK strongest overall in zero-shot TTS among the compared systems.AuK reduces average recognition error versus Qwen3-TTS from 3.07% to 2.65% and improves average SIM over Seed-TTS from 0.778 to 0.795.
- Speech generation: 83.37% Chinese accuracy is AuK’s best DSD result, while AuK-Flash reaches 82.40% on English DSD, tying Qwen3-TTS-VD for best performance.The full model outperforms Flash on all three Chinese metrics, whereas Flash has the stronger English DSD result.
- General speech editing: AuK leads broad editing evaluations, including 91.83% content-editing success and 71.33% prosody-editing success versus Ming-UniAudio’s 76.46% and 26.50%.AuK leads content, emotion, prosody, and acoustic editing on SpeechEditBench, while AuK-Flash leads paralinguistic editing with 39.25%.
- General speech editing: 3.09% Chinese and 3.96% English average WER under the Full semantic-editing setting improve on Ming-UniAudio’s 10.46% and 14.28%.Editing accuracy also rises to 91.47% in Chinese and 85.25% in English, with gains extending across deletion, insertion, and substitution.
8 Conclusion
AuK unifies speech generation and editing through an instruction-conditioned waveform interface, and AuK-Flash preserves broad capability with substantially lower inference cost. Experiments report leading generation and editing performance, competitive restoration results, and remaining limitations in unconstrained instruction understanding.
- AuK unifies speech generation and editing through a common instruction-conditioned waveform generation interface.
- 3.03 billion instruction–audio instances and 1.95 million hours of effective supervision span five task families.
- AuK-Flash provides four-step inference without classifier-free guidance and achieves a 4.5× wall-clock speedup.
- Experiments show leading performance on zero-shot and voice-design speech generation and general instruction-guided editing, with competitive speech restoration results.
- Robust native understanding of unconstrained editing requests remains incomplete, and the system still benefits from explicit task routing and prompt enhancement.
Contribution
The project received contributions spanning engineering, training, audio and music editing data, dialect speech data, infrastructure, design, sponsorship, and advisory support.
- Steve Yves, Shan Yang, Liefeng Bo, Zilong Zheng, Kai Yu, Eng-Siong Chng, and Xie Chen are listed as project sponsors and advisors.
- Support covered the VAE, sound generation and editing post-training, InstructTTS post-training, and the prompt enhancer.
- Audio editing data support was provided by Yalin Li, Jie Fang, Yifan Duan, and Zeyue Tian.
- Music editing data support was provided by Guangzheng Li, Haina Zhu, Shuyi Wang, and Jinwen Wang.
- Mingyu Cui supported dialect speech data, while Mingyu Cui, Tian Tan, Auden, and Sen Liang supported infrastructure and design.
A Detailed Evaluation Results
The appendix provides detailed AuK and AuK-Flash comparisons across speech generation, general instruction-guided editing, and signal-level enhancement and separation.
- The evaluation retains each benchmark’s native metrics while using broader baseline sets.
- The comparisons cover linguistic accuracy, speaker preservation, instruction adherence, and perceptual quality.
- AuK and AuK-Flash are evaluated in speech generation, general instruction-guided editing, and signal-level enhancement and separation.
A.1 Generation Benchmarks
The generation benchmarks evaluate zero-shot voice cloning and instruction-following text-to-speech across multiple languages and control dimensions.
- Seed-TTS-Eval measures zero-shot voice cloning across English, Chinese, and challenging Chinese text using recognition error and speaker similarity.
- InstructTTSEval measures control over acoustic parameters, descriptive styles, and role-playing instructions in Chinese and English.
- Both benchmarks report means over three runs using benchmark-specific instruction templates and duration estimation without prompt enhancement.
- Tables 5 and 6 present zero-shot TTS and instruction-following TTS performance, respectively.
A.2 General Speech Editing Benchmarks
This section evaluates general and signal-level speech editing across instruction following, semantic and acoustic edits, enhancement, separation, and super-resolution. The benchmarks combine task-specific measures of edit success, linguistic preservation, speaker identity, and perceptual quality.
- General instruction-guided editing: MMAE-Speech measures instruction following, content retention, and overall edit success with the Prompt Enhancer enabled.
- Semantic and acoustic editing: Ming-Freeform-Audio-Edit evaluates deletion, insertion, substitution, and acoustic control over speaking rate and pitch under Basic and Full settings.
- Benchmark coverage: Tables 7–9 report instruction-guided and semantic editing performance on MMAE-Speech and the Basic and Full Ming-Freeform-Audio-Edit splits.
- Evaluation measures: Evaluation combines perceptual-quality scores with WER, dWER, PER, and SIM to assess linguistic preservation and speaker-identity preservation.