Source-linked AI summary
VoiceSculptor: Your Voice, Designed By You
Jingbin Hu, Huakang Chen, Linhan Ma, Dake Guo, Qirui Zhan, Wenhao Li, Haoyu Zhang, Kangxiang Xia, Ziyu Zhang, Wenjie Tian, Chengyou Wang, Jinrui Liang, Shuhan Guo, Zihang Yang, Bengu Wu, Binbin Zhang, Pengcheng Zhu, Pengyuan Xie, Chuan Xie, Qiang Zhang, Jie Liu, Lei Xie
TL;DR
Open-source TTS still lacks flexible, fine-grained natural-language control over attributes such as pitch, rate, age, emotion, and style. VoiceSculptor addresses this gap with instruction-based voice design, structured attribute reasoning, retrieval grounding, and downstream voice cloning. It achieves open-source SOTA on InstructTTSEval-Zh, while the authors identify stability, latency, demographic coverage, and multilingual evaluation as remaining boundaries.
Problem
Open-source TTS systems provide limited natural-language control over fine-grained speech attributes such as pitch, speaking rate, age, emotion, and style.
Method
VoiceSculptor combines natural-language voice design, CoT-based fine-grained attribute modeling, RAG-based instruction grounding, and voice cloning in one framework.
Results
VoiceSculptor-VD achieves SOTA performance among open-source instruction-following TTS systems on InstructTTSEval-Zh, outperforming strong baselines across multiple metrics.
Takeaways & Limitations
The framework provides an open-source approach to instruction-controlled, fine-grained voice synthesis with reusable voice representations for downstream speech generation.
Takeaways & Limitations
The model remains unstable under repeated synthesis, can produce silence or delayed responses, performs insufficiently on elderly and child voices, and lacks comprehensive English or multilingual evaluation.
Abstract
from arXiv · showhide
Despite rapid progress in text-to-speech (TTS), open-source systems still lack truly instruction-following, fine-grained control over core speech attributes (e.g., pitch, speaking rate, age, emotion, and style). We present VoiceSculptor, an open-source unified system that bridges this gap by integrating instruction-based voice design and high-fidelity voice cloning in a single framework. It generates controllable speaker timbre directly from natural-language descriptions, supports iterative refinement via Retrieval-Augmented Generation (RAG), and provides attribute-level edits across multiple dimensions. The designed voice is then rendered into a prompt waveform and fed into a cloning model to enable high-fidelity timbre transfer for downstream speech synthesis. VoiceSculptor achieves open-source state-of-the-art (SOTA) on InstructTTSEval-Zh, and is fully open-sourced, including code and pretrained models, to advance reproducible instruction-controlled TTS research.
1 Introduction
VoiceSculptor addresses the limited fine-grained control of existing speech generation systems by designing voices from natural-language instructions. It combines structured attribute reasoning, retrieval-based instruction grounding, and voice cloning for controllable speech synthesis.
- Motivation: Existing speech systems generate natural speech and mimic timbre from reference audio, but provide limited direct control over fine-grained attributes.The constrained attributes include pitch, speaking rate, age, emotional expression, and speaking style.
- Motivation: Earlier controllable TTS methods relied on fixed prompts or limited latent spaces, restricting flexible instruction-driven voice design.
- Proposed Framework: VoiceSculptor enables users to design speaker timbre and manipulate multiple voice attributes through free-form natural-language instructions.The framework is presented as an open-source alternative to less transparent commercial systems.
- Proposed Framework: The voice design module decomposes natural-language instructions into intermediate attribute reasoning steps to control prosody, style, and speaker characteristics.These steps are represented as auxiliary attribute tokens that connect abstract descriptions with acoustic realizations.
- Proposed Framework: Retrieval-Augmented Generation retrieves relevant instruction examples and attribute knowledge to support iterative refinement and out-of-domain generalization.The designed voice can then provide a prompt waveform for downstream speech synthesis through the integrated voice cloning module.
2 Architecture
VoiceSculptor uses a language-model-based voice design module with discrete speech tokens, fine-grained attribute reasoning, and retrieval grounding, then connects the resulting voice representation to downstream synthesis. Its training pipeline builds multidimensionally annotated data, while joint supervision and token dropout support controllable attribute rendering.
- Architecture: VoiceSculptor combines LLaSA-3B for voice design with CosyVoice2 for voice cloning, using XCodec2 to represent speech as discrete audio tokens.The design model interprets instructions and predicts audio-token sequences that are decoded into speech waveforms.
- Architecture: The system’s contributions include large-scale annotated data processing, CoT-based attribute modeling, and RAG-based grounding for diverse instructions.Retrieved in-domain examples are supplied at inference time to ground instruction interpretation.
- Data Pipeline: The data pipeline denoises and filters speech, performs speech recognition and forced alignment, and adds multidimensional annotations for acoustic and speaker attributes.Annotations cover pitch, speaking rate, loudness, gender, age, emotion, paralinguistic characteristics, and context.
- Data Pipeline: The resulting annotations are regularized into instruction-like text representations and assembled into a dataset for natural-language voice control and attribute manipulation.The pipeline is designed to support robust instruction understanding through iterative collection, filtering, annotation, and validation.
- Fine-Grained Attribute Modeling: CoT-based attribute modeling structures acoustic information as intermediate semantic steps linking natural-language instructions to acoustic realizations.The model jointly represents instruction text, attribute tokens, and discrete speech tokens in an autoregressive framework.
- Training and Generalization: Joint cross-entropy supervision predicts linguistic, attribute, and speech tokens, coupling instruction semantics with acoustic generation.Attribute-token dropout, with probabilities such as 0.2, encourages inference from natural language and contextual cues rather than explicit attributes alone.
- Training and Generalization: RAG uses semantically related in-domain instructions to reduce sensitivity to distribution shifts and improve robustness for open-ended prompts.The retrieved examples support more consistent generation under unseen or structurally diverse instructions.
3 Experiments
VoiceSculptor is evaluated on instruction-following and controllability, scaling behavior, and component ablations. It achieves open-source state-of-the-art performance on InstructTTSEval-Zh, while CoT tokens, text CE loss, RAG, and larger models or datasets improve results.
- Evaluation on InstructTTSEval-Zh: VoiceSculptor-VD attains the highest overall AVG score among open systems, despite MiMo-Audio-7B-Instruct performing slightly better on DSD.The comparison indicates stronger APS and RP performance for VoiceSculptor-VD.
- Evaluation on InstructTTSEval-Zh: VoiceSculptor-VD achieves state-of-the-art performance among open-source instruction-following TTS systems on InstructTTSEval-Zh.It consistently surpasses open-source and instruction-tuned baselines across most metrics, with strongest performance on APS and RP.
- Scaling Study: Larger models and more diverse training data consistently improve speech instruction-following performance.Scaling from 1B to 3B parameters improves all metrics under identical SFT settings, while richer data and CPT initialization further boost Voice Design performance.
- CoT-based Attribute Tokens: CoT-based fine-grained attribute tokens consistently improve IMOS, APS, DSD, RP, and AVG without changing the model architecture.The gains generalize across model scales and support more precise, disentangled control of voice characteristics.
- Text Cross-Entropy Loss: A text-side CE loss improves all reported evaluation metrics by jointly supervising instruction and speech representations.The results indicate stronger alignment between textual instructions and generated speech, with improved context understanding and instruction following.
- RAG Ablation: RAG produces especially large gains of +7.1% in APS, +13.0% in RP, and +8.2% in AVG.Retrieved examples and structured attribute information improve instruction grounding, while the non-RAG gap exposes sensitivity to phrasing and unseen descriptions.
4 Limitations and Future Work
The model remains limited in practical stability, response latency, demographic coverage, and evaluation breadth. Future work targets stronger text understanding, broader training, improved representations, and more robust instruction handling.
- Current limitations: Repeated synthesis under the same instruction can fail to maintain precise control over desired attributes.The system may also produce long silences or delayed responses during synthesis and interaction.
- Current limitations: The model performs insufficiently on elderly and child voices, with inadequate naturalness and timbre consistency.
- Evaluation scope: Evaluation of instruction following is incomplete because existing benchmarks focus primarily on Chinese tasks.Comprehensive English and multilingual assessments have not yet been conducted.
- Future work: Future work will strengthen text understanding through large-scale text pretraining, instruction-data augmentation, and more expressive audio representations.The stated goal is to improve linguistic representations, robustness, and generalization.
5 Conclusion
VoiceSculptor is a unified natural-language-driven framework for fine-grained controllable speech synthesis. It achieves open-source SOTA on InstructTTSEval-Zh while its scaling and ablation studies support the value of its training and modeling choices.
- Overall conclusion: VoiceSculptor-VD achieves open-source SOTA on InstructTTSEval-Zh, consistently outperforming strong baselines across complementary metrics.
- Scaling and training: Instruction following benefits from increased model capacity, richer training data, and staged training with large-scale continual pre-training.
- Ablation findings: CoT-based attribute tokens, text-side cross-entropy supervision, and RAG each improve instruction understanding, controllability, and robustness.
- Ablation findings: CoT-based attribute modeling provides more precise and interpretable control over vocal characteristics.
6 Ethics Statement
The ethics statement restricts use of VoiceSculptor for unauthorized cloning, impersonation, fraud, scams, deepfakes, and other illegal or malicious activities. It also clarifies that generated voices are synthetic outputs rather than recordings or reproductions of identifiable individuals.
- Responsible use: VoiceSculptor should not be used for unauthorized voice cloning, impersonation, fraud, scams, deepfakes, or illegal or malicious activities.Users are instructed to comply with local laws and regulations and uphold ethical standards.
- Nature of outputs: Generated voices are synthetic outputs inferred by the model, not recordings of authentic human voices or reproductions of specific identifiable individuals.
7 Appendix
The appendix reports validation-loss ablations for CoT-based attribute tokens, model scale, and training-data configuration. It specifies the compared models, training resources, data settings, and model-selection procedure.
- CoT and model scale: Lower validation-loss curves correspond to models with CoT-based fine-grained attribute tokens, while higher curves represent baselines without them.Baseline attribute tokens are randomly dropped with probability 0.2 during training.
- CoT and model scale: The ablation compares 1B models trained on 8×L40 GPUs with 3B models trained on 8×A100 GPUs.All curves report validation loss as a function of training steps, using 3700 hours of data.
- Model selection: Each experiment selects epoch 3, the epoch with the lowest validation-set loss, as its final model.
- Training-data configuration: With identical 3B architectures and optimization settings, the data configurations are 1000h SFT, 3700h SFT, and 9k h continued pretraining followed by 3700h SFT.