PaperlayerView exact revision on arXiv

Source-linked AI summary

HQ-SVC: Towards High-Quality Zero-Shot Singing Voice Conversion in Low-Resource Scenarios

Bingsong Bai, Yizhong Geng, Fengping Wang, Cong Wang, Puyuan Guo, Yingming Gao, Ya Li

arXiv:2511.08496v3cs.SDcs.AIeess.AS

TL;DR

Zero-shot singing voice conversion must preserve melody and content while converting to an unseen singer, but existing methods face information-loss and resource challenges. HQ-SVC combines decoupled codec features with additional acoustic modeling and DDSP/diffusion refinement. The authors report strong zero-shot conversion results and performance in voice super-resolution.

  • Problem

    Zero-shot SVC seeks to convert a singer’s timbre to an unseen target while preserving melody and content, but separate content and speaker encoders can lose information and existing methods can require substantial resources.

    Source: Introduction · p. 1, Zero-shot SVC · p. 2

  • Method

    HQ-SVC extracts content and speaker features with a decoupled codec, adds acoustic features through EVA, and refines reconstruction with DDSP and diffusion.

    Source: Introduction · pp. 1–2, Methods · p. 2

  • Results

    HQ-SVC achieves high naturalness and similarity in zero-shot singing voice conversion and performs well in zero-shot voice super-resolution, while outperforming SaMoye across several reported measures.

    Source: Zero-shot Singing Voice Conversion… · p. 6, Conclusions · p. 7

  • Takeaways & Limitations

    The reported results show that HQ-SVC supports zero-shot singing voice conversion and voice super-resolution with limited GPU resources and a small training dataset.

    Source: Conclusions · p. 7

Abstract

from arXiv · show

Zero-shot singing voice conversion (SVC) transforms a source singer's timbre to an unseen target speaker's voice while preserving melodic content without fine-tuning. Existing methods model speaker timbre and vocal content separately, losing essential acoustic information that degrades output quality while requiring significant computational resources. To overcome these limitations, we propose HQ-SVC, an efficient framework for high-quality zero-shot SVC. HQ-SVC first extracts jointly content and speaker features using a decoupled codec. It then enhances fidelity through pitch and volume modeling, preserving critical acoustic information typically lost in separate modeling approaches, and progressively refines outputs via differentiable signal processing and diffusion techniques. Evaluations confirm HQ-SVC significantly outperforms state-of-the-art zero-shot SVC methods in conversion quality and efficiency. Beyond voice conversion, HQ-SVC achieves superior voice naturalness compared to specialized audio super-resolution methods while natively supporting voice super-resolution tasks.

Introduction

HQ-SVC targets zero-shot singing voice conversion by combining decoupled content and speaker features with additional acoustic modeling and progressive synthesis refinement. It is designed for low-resource settings and also supports voice super-resolution.

Introduction

  • Existing zero-shot SVC methods face a trade-off between synthesis quality and resource demands, while separate content and speaker encoders can lose information.FastSVC has poor synthesis quality; LDM-SVC and SaMoye-SVC require substantial computation, training time, or data.Source: Introduction · p. 1
  • HQ-SVC uses a decoupled codec to extract content and speaker features, then adds acoustic features and refines synthesis with DDSP and diffusion.The EVA module supports multi-feature fusion and includes speaker loss and a Speaker-F0 Predictor.Source: Introduction · pp. 1–2
  • HQ-SVC is designed for low-resource conversion of unseen speakers without unnecessary retraining or adversarial training.The paper describes training on small datasets and a single consumer-grade GPU.Source: Introduction · p. 2
  • HQ-SVC outperforms the current state-of-the-art method across most objective and subjective evaluation metrics and reports higher voice naturalness and speaker similarity than the current state-of-the-art audio SR method.The framework also extends to zero-shot singing and speech super-resolution.Source: Introduction · p. 2

Related Works

Prior singing voice conversion research progressed from parallel-data methods to GAN-based and low-resource approaches, but zero-shot conversion still faces generalization and resource challenges. HQ-SVC is discussed against this background of quality, training cost, and unseen-speaker conversion.

Singing Voice Conversion

  • Early SVC methods used Gaussian Mixture Models and required source and target singers to perform the same song, while GAN-based methods removed the parallel-data requirement but could be harder to train.SoVITS-SVC added a flow-based model to address mode collapse but still required considerable computational resources.Source: Singing Voice Conversion · p. 2

High-Quality SVC in Low-resource Scenarios

  • Low-resource SVC seeks high-quality output with little singing data and limited GPU power, motivating DDSP- and diffusion-based methods.The cited approaches are described as offering generation quality and training stability under low-resource conditions.Source: High-Quality SVC in Low-resource… · p. 2

Zero-shot SVC

  • Zero-shot SVC converts without target-singer data or fine-tuning, but existing methods face unseen-speaker generalization and training-resource challenges.FastSVC is lightweight but its generation quality leaves room for improvement; higher-quality VITS-based approaches require considerable training computation.Source: Zero-shot SVC · p. 2

Methods

HQ-SVC combines codec-derived content and speaker features with acoustic conditioning, then refines singing reconstruction through DDSP and diffusion. Its EVA module adds pitch, energy, and phase information to the feature-fusion process.

Methods

  • HQ-SVC uses decoupled codec features, adds pitch and energy through EVA, and optimizes reconstruction against high-sampling-rate Mel spectrograms for conversion and voice super-resolution.The architecture is built on a decoupled audio codec that provides content features and speaker embeddings.Source: Methods · pp. 2–3

Decoupled Codec

  • Frozen FACodec encoder and decoder layers provide the content and speaker features used by HQ-SVC.FACodec is used as the unified disentangler.Source: Decoupled Codec · p. 3

DDSP Model

  • DDSP combines classical signal processing with deep learning, and its harmonic and noise synthesizers produce periodic and aperiodic waveform components.Its modular design allows control over attributes such as timbre and loudness.Source: DDSP Model · p. 3

Diffusion Model

  • A diffusion model processes DDSP output to enrich sound-characteristic descriptions and improve SVC performance.The paper points to DiffSVC and CoMoSVC as prior demonstrations of diffusion models in SVC.Source: Diffusion Model · p. 3

Enhanced Voice Adaption Module

  • EVA adds RMVPE-derived pitch, energy, and phase features for multi-feature fusion because melody and energy dynamics challenge synthesis from content and speaker features alone.The phase features are included for the DDSP model.Source: Enhanced Voice Adaption Module · p. 3

Acoustic Feature Extraction and Mapping

  • Speaker features are mapped to speaker embeddings, while pitch, volume, and phase features are mapped to their respective embeddings.The residual between speaker features and speaker embeddings forms style features; the F0 transformation is intended to help capture melody.Source: Acoustic Feature Extraction and Mapping · p. 3

Fusion of Style and Content Embeddings

  • Style combines speaker and F0 embeddings, then FiLM fuses it with content; speaker-contrastive loss and SFP predict pitch statistics from speaker embeddings.The speaker loss pulls positive samples closer and pushes negative samples farther within a batch.Source: Fusion of Style and Content Embeddings · pp. 3–4

Singing Voice Reconstruction

  • The EVA embedding passes through DDSP to generate audio and a Mel spectrogram, then diffusion further optimizes the reconstruction.DDSP uses an MSE loss between its predicted and real Mel spectrograms; total HQ-SVC loss combines DDSP, diffusion, speaker, and F0 losses.Source: Singing Voice Reconstruction · p. 4

Datasets

Training used two Mandarin singing datasets, with held-out Opensinger singers reserved for unseen-speaker testing. Audio was standardized and processed into acoustic and speaker features.

Training Datasets

  • Opensinger and M4Singer provided the Mandarin singing data used for training.Two male and two female Opensinger singers were held out from training and validation for testing.Source: Training Datasets · p. 4
  • One percent of the remaining data formed validation, and clips shorter than 2.1 seconds were excluded.The validation subset monitored convergence; short clips were removed to avoid potential noise and performance degradation.Source: Training Datasets · p. 4

Data Preprocessing

  • Preprocessing resampled audio to 44.1 kHz, extracted Mel and energy features, and used 16 kHz audio for pitch and FACodec features.The pipeline used 128-dimensional Mel spectrograms, RMVPE pitch features, and 256-dimensional content and speaker features.Source: Data Preprocessing · p. 4

Experimental Setups

The setup trained HQ-SVC on a single GPU and evaluated unseen-speaker conversion and voice super-resolution with objective and subjective measures. The protocols covered singing and speech datasets and included comparison with established baselines.

Model Training

  • 250k training steps on one RTX 3090 used less than 6 GB of GPU memory and took about 11 hours.Training used AdamW with a batch size of 64.Source: Model Training · p. 5

Model Evaluation on Zero-shot Singing Voice Conversion

  • Zero-shot conversion compared HQ-SVC trained on under 80 hours with SaMoye-SVC pretrained on 1700 hours and a fine-tuned FACodec baseline.Audio was resampled to 32 kHz for evaluation; measures covered intelligibility, timbre similarity, pitch accuracy, quality, naturalness, and speaker similarity.Source: Model Evaluation on Zero-shot Singing… · p. 5
  • Conversion tests used 128 samples for objective evaluation and 96 samples per model for subjective ratings from 12 volunteers.The test used four samples from two male and two female unseen Opensinger singers, with cross-conversion between source and target samples.Source: Model Evaluation on Zero-shot Singing… · p. 5

Model Evaluation on Voice Super-resolution

  • Voice super-resolution compared HQ-SVC with AudioSR on unseen Mandarin and English singing and speech datasets.HQ-SVC output was set to 44.1 kHz; AudioSR uses an LDM trained on over 7,000 hours and can upscale audio to 48 kHz.Source: Model Evaluation on Voice… · p. 5
  • Super-resolution testing sampled 8 male and 8 female speakers across four datasets, with 128 samples for objective evaluation.Subjective evaluation used 96 singing and 96 speech samples per model; measures included STOI, SECS, FPC, NISQA, LSD, NMOS, and SMOS.Source: Model Evaluation on Voice… · pp. 5–6

Experimental Results and Analysis

Across singing voice conversion, HQ-SVC outperforms FACodec-SVC and reports advantages over SaMoye on multiple measures, while voice super-resolution results show trade-offs against AudioSR. Sampler and ablation analyses identify a speed-quality operating point and complementary roles for diffusion and DDSP.

Zero-shot Singing Voice Conversion Comparison

  • HQ-SVC outperforms FACodec-SVC across all metrics and exceeds SaMoye on several objective and subjective measures, despite SaMoye’s SECS advantage.HQ-SVC maintains advantages in STOI, F0 RMSE, FPC, NISQA, NMOS, and SMOS; its timbre similarity also remains superior to other baselines.Source: Zero-shot Singing Voice Conversion… · p. 6
  • SECS favors SaMoye while SMOS slightly favors HQ-SVC; adding pitch and speaker losses improves pitch and naturalness but can make pitch-consistent voice changes harder for listeners to identify.The authors attribute the metric difference partly to pitch cues in CAM++ speaker embeddings and listeners’ sensitivity to pitch when judging speaker identity.Source: Zero-shot Singing Voice Conversion… · p. 6
  • HQ-SVC’s F0 contour is closest to the source, while SaMoye’s is overly smooth and FACodec-SVC’s fluctuates; the latter systems also show harmonic smoothing or added noise.These spectrogram observations align with the reported F0 RMSE and FPC results.Source: Zero-shot Singing Voice Conversion… · p. 6

Voice Super-resolution Comparison

  • For voice super-resolution, HQ-SVC leads AudioSR in LSD and NISQA, while AudioSR leads in STOI and FPC and better preserves pronunciation and F0.The comparison uses Table 2; the authors attribute AudioSR’s advantage on pronunciation and F0 partly to its training on paired low- and high-sample-rate Mel spectrograms.Source: Voice Super-resolution Comparison · pp. 6–7, Table 2 · p. 6
  • HQ-SVC outperforms AudioSR in NMOS and SMOS; its spectra show clearer mid-to-high harmonics and less high-frequency noise.The subjective comparison is reported in Table 3, while the spectral comparison is shown in Figure 3.Source: Figure 3 · p. 6, Voice Super-resolution Comparison · p. 7, Table 3 · p. 7

Sampler and Acceleration Rate Comparison in HQ-SVC

Ablation Studies

  • Removing diffusion slightly clarifies speech but harms timbre and naturalness; removing DDSP raises some scores yet weakens timbre conversion because diffusion retains the original Mel.The ablation comparison is reported in Table 5.Source: Ablation Studies · p. 7, Table 5 · p. 7
  • The single encoder fuses information and controls rhythm better than separate encoders, but HQ-SVC trails HQ-SVC-SE in timbre matching, motivating improved speaker-identity separation.The authors characterize HQ-SVC as balancing performance across measures rather than leading every individual area.Source: Ablation Studies · p. 7

Conclusions

HQ-SVC is a zero-shot singing voice conversion method for low-resource scenarios, using a decoupled codec and Enhanced Voice Adaption to combine content, speaker, and additional acoustic features. Objective and subjective experiments report high naturalness and similarity for conversion and good performance on zero-shot voice super-resolution despite limited GPU resources and a small training dataset.

Conclusions

  • HQ-SVC combines decoupled-codec content and speaker features with additional acoustic features through the Enhanced Voice Adaption module.The paper presents the framework for low-resource zero-shot singing voice conversion.Source: Conclusions · p. 7
  • Objective and subjective experiments report high naturalness and similarity in zero-shot singing voice conversion.Source: Conclusions · p. 7
  • HQ-SVC also performs well in zero-shot voice super-resolution despite limited GPU resources and a small training dataset.Future work will focus on singing voice style conversion.Source: Conclusions · p. 7
Loading 2511.08496v3…