Source-linked AI summary
Vocal Intensity Conversion with Continuous SPL Control via Adversarial Training
Quentin Le Tellier, Albert Rilliard, Olivier Perrotin, Marc Evrard
TL;DR
Existing vocal-intensity conversion systems rely on reference utterances or discrete categories, while this paper controls speech with a continuous dB SPL target. It trains a conditional cycle-consistent converter in frozen speech-feature space and evaluates achieved level; WavLM reaches 83% of the target intensity change, with perceived level increasing as the target rises.
Problem
Existing systems control vocal intensity through a reference utterance or a few discrete categories rather than a continuous physical SPL target.
Source: §1 · p. 1Method
A conditional cycle-consistent GAN converts frozen speech-encoder features using a continuous dB SPL target, without parallel data.
Source: §1 · p. 1Results
WavLM L6 reaches 0.83 of the target vocal-intensity displacement, and perceived level increases with the target level.
Source: §4 · p. 4Takeaways & Limitations
A scalar dB SPL target provides a defined control value within the training range without carrying properties of a reference utterance into the output.
Source: §1 · p. 1The target level requires a calibrated recording chain and a known mouth-to-microphone distance so pressure can be referred to 1 m.
Source: §2.1 · p. 1
Abstract
from arXiv · showhide
Vocal intensity conversion renders a recorded utterance at a different vocal intensity, rather than simply scaling its amplitude, while preserving its content and speaker identity. Existing systems control intensity through a reference utterance or a few discrete categories. Instead, we control the conversion with any target sound pressure level (SPL) on a continuous dB scale. A lightweight converter, trained as a conditional cycle-consistent GAN without parallel data, transforms the features of a frozen speech encoder, from which a pretrained vocoder resynthesizes the speech. We compare three representations, log-mel spectrograms, WavLM, and SpeechTokenizer, and introduce an objective protocol that measures the achieved intensity change against the target one on a calibrated corpus. With WavLM, the converter achieves a measured 83% of the target change in intensity and also shifts the fundamental frequency. A perceptual evaluation confirms that perceived vocal intensity increases with the target level on amplitude-normalized stimuli.
1. INTRODUCTION
The paper frames vocal-intensity conversion as rendering speech at a chosen physical SPL while preserving content and speaker identity. It replaces reference-utterance or categorical control with continuous dB SPL conditioning and evaluates whether the requested level is achieved.
1. INTRODUCTION
- Existing normal-to-Lombard systems inherit discrete noise or effort categories from their corpora, including latent swapping and lookup-table control.LombardTokenizer replaces a codec layer with an encoded target utterance, whereas PGD-N2L injects a selected noise-level embedding into the content stream.Source: §1 · p. 1
- Continuous dB SPL conditioning replaces reference-utterance or noise-category targets, while a frozen encoder and decoder leave only a lightweight converter to train.The target is defined for any value within the training range and carries no reference utterance properties into the output.Source: §1 · p. 1
- A frozen intensity predictor reports each conversion’s achieved level relative to its target, enabling physical-unit evaluation unavailable to the two closely related systems.The predictor is used for evaluation and is absent from the generator objective.Source: §1 · p. 1
- The paper contributes a conditional cycle-consistent GAN trained without parallel data or an auxiliary intensity predictor, comparisons across three frozen encoder/decoder pairs, and objective and perceptual evaluation protocols.The motivation includes targeted data augmentation and expressive-synthesis or cognitive-evaluation control; speech technologies are reported to underperform outside conversational effort.Source: §1 · p. 1
2. METHOD
The method conditions feature-space conversion on a continuous, physically defined vocal-intensity target and trains a converter while keeping the encoder and decoder frozen. Its conditional adversarial objective uses real and mismatched sequence-level pairs alongside cycle consistency and an identity anchor.
2.1. Task formulation
- A continuous target level is well posed only when the recording chain is calibrated and mouth-to-microphone distance is known, allowing pressure to be referred to 1 m.These conditions make levels comparable across sessions and speakers.Source: §2.1 · p. 1
2.2. Conversion in a frozen feature space
- The converter transforms encoder frames while the encoder and decoder remain frozen, so output reconstruction quality depends on the chosen pair.It emits one frame per input frame and uses a residual correction; duration and speaking rate cannot change, though a per-frame target could represent an intensity contour.Source: §2.2 · p. 2
2.3. τ-conditional training objective
- The conditional discriminator scores sequence-level pairs, separating real sequences paired with their own levels from converted pairs and real sequences paired with mismatched levels.Conditioning the discriminator itself targets the joint sequence-and-level distribution rather than relying on an auxiliary predictor to set the relation.Source: §2.3 · p. 2
- For continuous levels, nearby levels within v of a real sequence’s source level count as positive pairs, while mismatched levels are at least δ > v away.Cycle consistency requires conversion back at the source level to recover the original frames, and an identity anchor requires unchanged output when target and source levels match.Source: §2.3 · p. 2
3. EXPERIMENTAL SETUP
The experiments use calibrated AVID speech and three frozen encoder/decoder pairs to train and compare a converter without parallel training targets. Objective displacement and preservation measures complement a listener study of perceived intensity.
3.1. Corpus
- AVID provides calibrated recordings from 50 speakers reading 25 sentences and two paragraphs at four intensity categories; speakers are split 40/4/6, and training uses no parallel data.The categories span roughly 40 dB, with speaker-disjoint, sex-stratified splits.Source: §3.1 · p. 2
3.2. Encoder/decoder pairs
- The comparison uses frozen log-mel/BigVGAN-v2, WavLM-Large layer 6/HiFi-GAN, and continuous SpeechTokenizer encoder outputs.The same converter architecture, objective, and corpus are used across the three pairs.Source: §3.2 · pp. 2–3
3.3. Training
- Converters train on 2-second excerpts with target levels sampled uniformly from 42.8–76.1 dB SPL for 11,400 iterations.Training takes about 3.7 hours for WavLM L6 and SpeechTokenizer and about 9 hours for log-mel.Source: §3.3 · p. 3
3.4. The intensity predictor
- The frozen intensity predictor is a wav2vec 2.0 regression head, evaluated on six unseen speakers with 1.43 dB RMSE and +0.47 dB mean error.Its input is normalized per utterance, removing amplitude as a cue.Source: §3.4 · p. 3
3.5. Evaluation protocol
- Objective evaluation converts recordings within speaker-and-sentence groups to measured target levels, then regresses achieved displacement on target displacement; a slope of 1 means full realization.The protocol also measures acoustic changes and separately reports WER, predicted naturalness, and speaker similarity.Source: §3.5 · p. 3
3.6. Perceptual evaluation protocol
- The listener study peak-normalizes stimuli and estimates perceived levels by probit regression against measured-level anchors, with confidence intervals resampled across listeners.It uses 64 converted stimuli, and listener retention is based on control-pair accuracy.Source: §3.6 · pp. 3–4
4. RESULTS
WavLM L6 best matches the target intensity displacement among the tested representations, while different feature spaces control different acoustic attributes. Listeners perceive increasing intensity as targets rise, but absolute perceived level varies by source recording.
4. RESULTS
- 0.83 for WavLM L6, versus 0.62 for log-mel and 0.58 for SpeechTokenizer, on predicted vocal-intensity displacement; WavLM L6 ranks best against each representation’s ceiling.The displacement slopes are reported in Table 1.Source: §4 · p. 4
- 0.48 for WavLM L6 and 0.42 for log-mel on f0 displacement, while log-mel leads on alpha ratio (0.75) and F1 (0.65); no feature space leads on every measure.SpeechTokenizer barely displaces f0 (0.13) despite a 0.94 ceiling.Source: §4 · p. 4
- 0.61 speaker similarity for WavLM L6 conversions, versus 0.73 between original recordings at source and target levels; conversion also increases WER and lowers predicted naturalness for every representation.WavLM L6 has the lowest converted WER, while SpeechTokenizer has the highest UTMOS.Source: §4 · p. 4
- 0.69 (95% interval 0.60–0.78): perceived level rises with target level across the 30 dB target range, which listeners perceive as a 21 dB range.Original recordings have a 0.68 slope (0.51–0.84), not different from conversions.Source: §4 · p. 4, Figure 2 · p. 4
- About 6 dB standard deviation in perceived level across source recordings at the same target means relative intensity is controlled better than absolute perceived level.Within each source recording, conversions are perceived in target order, with rank correlations from 0.62 to 0.95.Source: §4 · p. 4
5. CONCLUSION
The conclusion frames vocal-intensity conversion as continuous dB SPL conditioning with a conditional cycle-consistent GAN over frozen features. WavLM L6 reaches 83% of the target intensity change, while perceived level rises with target level.
5. CONCLUSION
- WavLM L6 achieves 83% of the target intensity change, and perceived vocal level rises with target level.Source: §5 · p. 4
- The system conditions on a continuous dB SPL target and uses a conditional cycle-consistent GAN over a frozen encoder/decoder feature space.Source: §5 · p. 4
- Control is more consistent relative to the source than at an absolute perceived level; conversions also displace f0 and spectral measures less than original recordings.Source: §5 · p. 4
- Calibrated corpora spanning wide vocal-intensity ranges are scarce, limiting available training data.Source: §5 · p. 4
- Future work could combine feature-space strengths with a hybrid set and use automatic source-level estimates to train on larger uncalibrated corpora.Source: §5 · p. 4