PaperlayerView exact revision on arXiv

Source-linked AI summary

StyleStream 2.0: Fast and Controllable Streaming Voice Style Conversion

Yisi Liu, Nicholas Lee, Gopala K. Anumanchipalli

arXiv:2610.28940v1cs.SDcs.AI

TL;DR

StyleStream 1.0 established streaming zero-shot VSC, but its roughly one-second latency and reliance on target reference utterances limited speed and text-based control. StyleStream 2.0 combines a few-step Stylizer and a unified text-conditioned style-embedding model for voice design and editing. It reports stronger target-style fidelity across VSC, voice design, and editing, with competitive intelligibility and preservation.

  • Problem

    Streaming VSC has faced roughly one-second latency, while existing systems largely require a target reference and prior text-guided speech-to-speech methods typically address only one style axis.

    Source: §1 · p. 1

  • Method

    StyleStream 2.0 uses a few-step pixel MeanFlow Stylizer with autoregressive feedback and a unified text-conditioned model that predicts or edits style embeddings.

    Source: §1 · p. 2

  • Results

    StyleStream 2.0 reports stronger target-style fidelity across VSC, voice design, and editing while remaining competitive in intelligibility and preservation.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The system combines streaming holistic VSC with natural-language voice design and single-attribute editing in a shared style-embedding space.

    Source: §1 · p. 2, §6 · p. 9

    The 520 ms latency still leaves room for tightly interactive use, and editing can drift on non-targeted attributes.

    Source: D Limitations · pp. 22–23

Abstract

from arXiv · show

Voice style conversion (VSC) aims to transform a source utterance to match the timbre, accent, and emotion of a target voice while preserving linguistic content. StyleStream 1.0 introduced the first streamable zero-shot VSC system with state-of-the-art conversion quality, but remained limited in two ways: approximately 1s end-to-end latency on consumer hardware, and the lack of a text-based interface for designing or editing target voices. We present StyleStream 2.0, a fast and controllable streaming VSC framework that addresses these limitations. To reduce latency, we train a few-step pixel MeanFlow model and further fine-tune it with autoregressive feedback inspired by Self Forcing, improving robustness under streaming inference. This reduces end-to-end latency to 520~ms on a consumer GPU, a 2x speedup over StyleStream 1.0, while maintaining conversion quality. To enable controllable style manipulation, we remove the mel-spectrogram context and instead route target style information through a compact style embedding. In this embedding space, we train a unified text-conditioned flow matching model that supports both text-based voice design, which maps natural language prompts to voice styles, and instruction-based voice editing, which modifies a specific attribute of a source utterance while preserving the rest. Experiments show that StyleStream 2.0 achieves the strongest target style fidelity on VSC, voice design, and voice editing, while remaining competitive on intelligibility and non-edited attribute preservation.

1 Introduction

StyleStream 2.0 targets two gaps in holistic streaming voice style conversion: high latency and the lack of text-based voice design and editing. It combines a faster Stylizer with a unified text-conditioned style-embedding model, reporting strong fidelity across conversion and text-control tasks.

1 Introduction

  • Prior VSC systems largely require a target reference utterance, making voice specification indirect, while prompt-controlled TTS does not convert an input utterance.Prior text-guided speech-to-speech work typically addresses a single axis, and no prior system provides holistic text-based design and editing in streaming VSC.Source: §1 · pp. 1–2
  • StyleStream 2.0 retains StyleStream 1.0’s two-stage architecture, reusing its Destylizer and focusing changes on the Stylizer.The Destylizer extracts content features disentangled from style; the Stylizer reconstructs speech from those features and a target style.Source: §1 · p. 2
  • 520 ms end-to-end latency on a consumer GPU is a 2× speedup over StyleStream 1.0 while maintaining conversion quality.The Stylizer uses few-step pixel MeanFlow and autoregressive feedback fine-tuning for chunk-by-chunk inference robustness.Source: §1 · p. 2
  • The text-conditioned model predicts a style embedding from a natural-language prompt for voice design, or modifies an existing embedding according to an instruction for editing.Removing mel-spectrogram context routes target style through a single embedding.Source: §1 · p. 2
  • Text-based control targets age, gender, accent, and emotion for prompt-based voice design and attribute-targeted editing.The paper presents these attributes as a structured, measurable interface for holistic VSC.Source: §1 · p. 2
  • StyleStream 2.0 shows strong target-style fidelity across reference-conditioned VSC, voice design, and editing, while remaining competitive in intelligibility, preservation, and audio quality.The comparison includes reference-based and cascaded baselines.Source: §1 · p. 2

2 Related Work

Prior work includes speaker-identity conversion, streaming conversion, accent and emotion conversion, holistic style transfer, and prompt-controlled TTS. StyleStream 2.0 addresses a remaining combination: fast streaming holistic VSC with unified text-based voice design and editing.

2 Related Work

  • Prior speech-to-speech work spans speaker-identity conversion, streaming or real-time conversion, and separate accent and emotional conversion tasks.More recent systems model holistic style across timbre, accent, and emotion.Source: §2 · pp. 2–3
  • Prompt-controlled TTS supports natural-language voice specification, while text-guided speech-to-speech methods typically focus on one attribute and are evaluated offline.The paper notes that only a few open-source TTS systems explicitly support holistic control over timbre, accent, and emotion.Source: §2 · p. 3
  • Few-step generative models reduce sampling cost, but short streaming chunks can still degrade quality at chunk boundaries.StyleStream 2.0 combines a few-step pixel MeanFlow Stylizer with autoregressive fine-tuning to improve streaming robustness.Source: §2 · p. 3

3 Method

StyleStream 2.0 retains StyleStream 1.0’s Destylizer–Stylizer framework: content features and a target-style embedding produce speech, with style supplied by a reference or text-conditioned Voice Controller. The method redesigns the Stylizer for few-step generation and streaming, and adds a unified controller for voice design and editing.

3.1 Destylizer

  • The reused Destylizer learns content-preserving features through ASR training and a compact FSQ bottleneck, then supplies its continuous pre-FSQ features at inference.It uses a frozen HuBERT-Large encoder, six conformer blocks, and an ASR decoder; the FSQ codebook has size 45.Source: §3.1 · p. 3

3.2.1 Preliminary: Pixel MeanFlow (pMF)

  • Pixel MeanFlow combines clean-sample prediction with interval-averaged velocity learning, converting its x-like prediction into a velocity for few-step guided generation.With CFG distillation, guided sampling requires one model evaluation per step.Source: §3.2.1 · p. 4

3.2.2 Stylizer Architecture and Training

  • The redesigned DiT removes mel-spectrogram context, making the style embedding the sole target-style input, and replaces the TDNN style encoder with a perceiver resampler.The resampler compresses concatenated WavLM layer outputs into a fixed-size representation used to form the style embedding.Source: §3.2.2 · p. 4

3.2.3 Autoregressive Fine-Tuning

  • Autoregressive fine-tuning trains the Stylizer on sequential chunk rollouts with generated mel history and future-content lookahead, matching streaming inference behavior.The style encoder is frozen; the loss covers only the current chunk, and generated past chunks are detached.Source: §3.2.3 · pp. 5–6

3.3 Voice Controller: Unified Voice Design and Editing

  • The text-conditioned Voice Controller generates style embeddings from prompts for voice design or modifies source style embeddings for instructed edits across age, gender, accent, and emotion.Design uses a learned null auxiliary embedding, while editing uses the patchified source style embedding.Source: §3.3 · p. 6

4 Experimental Setup

The experiments use separate reference-conditioned conversion and text-controlled design/editing evaluations, with task-specific data, baselines, and metrics.

4.1.1 Voice Style Conversion

  • The Stylizer is trained on roughly 50k hours of diverse English speech, and reference-conditioned evaluation pairs 300 source utterances with 10 targets for 3,000 test pairs.Targets cover five emotions and five accents.Source: §4.1.1 · p. 6

4.1.2 Voice Design and Editing

  • Voice Controller training uses CapSpeech-derived natural and synthetic data, while testing includes 3,000 design prompts and 3,000 single-attribute edits on the same 300 sources.Editing targets differ from source labels on the requested axis, allowing evaluation of edit success and preservation of other attributes.Source: §4.1.2 · pp. 6–7

4.2 Baselines

  • Table 1 presents voice style conversion results, marking the best and second-best results and treating ground-truth NMOS as an unranked upper bound.The caption identifies SS as StyleStream.Source: Table 1 · p. 7
  • Reference-conditioned conversion is compared with CosyVoice, Vevo, and StyleStream systems; text control uses ASR-plus-prompted-TTS design cascades and text-to-reference adapters for editing.StyleStream 2.0 is evaluated offline and in streaming mode.Source: §4.2 · p. 7

4.3 Metrics

  • Conversion metrics include WER, speaker/accent/emotion similarity, and naturalness; text-control metrics add attribute accuracy, non-edited-attribute preservation, prompt consistency, and editing preservation.The reported measures include NMOS and task-specific subjective MOS scores.Source: §4.3 · p. 7

5 Results

StyleStream 2.0 improves target-style fidelity across reference-conditioned conversion, voice design, and editing, while remaining competitive in intelligibility and preservation. Its results also show faster streaming and design choices that support chunked inference and text-based control.

5.1 Voice Style Conversion

  • StyleStream 2.0 ranks first on S-SIM, A-SIM, S-SMOS, and E-SMOS, improving over StyleStream 1.0 on most style metrics with 4 sampling steps.NMOS is statistically comparable across systems; StyleStream 2.0 remains competitive in WER while Vevo 2.0 has the lowest WER and lower target-style similarity across all metrics.Source: §5.1 · pp. 7–8

5.2 Streaming Analysis

  • 520.7 ms end-to-end latency comes from 300 ms chunks, 80 ms lookahead, and 140.7 ms per-chunk processing; StyleStream 2.0 runs in real time across the chunk-size sweep.At every tested chunk size, it dominates StyleStream 1.0 on nearly every metric; StyleStream 1.0 meets real-time processing only at 600 ms, with 598 ms processing time.Source: §5.2 · p. 8, Table 2 · p. 8

5.3 Voice Design and Editing

  • StyleStream 2.0 achieves the strongest overall voice-design style fidelity, with the highest age, gender, and accent accuracy among the cascaded baselines.Its streaming variant also improves age and gender accuracy over the baselines; CapTTS is competitive on accent control, while Qwen3-TTS has the best intelligibility and emotion accuracy but much lower accent accuracy.Source: Table 3 · p. 8, §5.3 · p. 9
  • StyleStream 2.0 offline has the highest edited-axis accuracy, lowest WER, and highest naturalness and style-consistency MOS in voice editing.The streaming variant remains close in style accuracy and SC-MOS; higher preservation scores for Vevo and Vevo 2 should be read alongside their lower edit success.Source: §5.3 · p. 9, Table 4 · p. 9

5.4 Ablation Studies

  • Without self-forcing, streaming intelligibility falls substantially while target-style similarity stays essentially unchanged, consistent with autoregressive rollout addressing chunk-inference train-test mismatch.Replacing the perceiver-resampler with the StyleStream 1.0 TDNN style encoder degrades intelligibility and accent similarity under the same offline pixel MeanFlow objective.Source: §5.4 · p. 9, Table 5 · p. 9

6 Conclusion

StyleStream 2.0 combines a few-step pixel MeanFlow Stylizer and autoregressive fine-tuning for faster streaming, while a unified text-conditioned controller supports voice design and single-attribute editing. Across reference-conditioned conversion, design, and editing, it improves target-style fidelity while remaining competitive in intelligibility and preservation.

6 Conclusion

  • StyleStream 2.0 combines a few-step pixel MeanFlow Stylizer with autoregressive fine-tuning and a unified text-conditioned controller for natural-language voice design and single-attribute editing.The conclusion reports improved target-style fidelity across reference-conditioned VSC, voice design, and editing, with competitive intelligibility and preservation.Source: §6 · p. 9

A Extended Data Generation and Evaluation Details

A.1 Training Corpus for Text-Based Style Control

The text-conditioned style-control corpus is built by extracting and normalizing speaker attributes, forming nested inventories, sampling source–target pairs, and rendering text supervision from templates. Caption labels and waveform predictions are combined under confidence rules, with waveform predictions used as fallback labels when captions lack trusted values.

A.1 Training Corpus for Text-Based Style Control

  • Caption attributes are extracted explicitly, with accent and emotion copied verbatim before deterministic normalization; VoxProfile predictions serve only as fallbacks when captions lack trusted values.The workflow also filters and normalizes attributes before forming nested utterance inventories and sampling source–target pairs.Source: §A.1 · p. 15
  • The training taxonomy buckets age into five ranges, uses specified accent and emotion vocabularies, and applies confidence thresholds when waveform-derived labels are used.Sex, accent, and emotion fallback thresholds are respectively 0.99, 0.8 with a 0.4 margin, and 0.7 with a 0.35 margin; source utterances are limited to 2–20 seconds and predicted ages 10–90.Source: §A.1 · pp. 15–16
  • The nested inventories contain 496,954 voice-conversion rows, 403,013 generation rows, and 59,613 editing rows, with progressively stricter attribute-label requirements.The editing inventory requires age, sex, accent, and emotion labels; the generation inventory additionally requires either an accent or emotion label.Source: §A.1 · p. 16
  • Training pairs hold content fixed for voice design, while accent and emotion editing pairs preserve the other attribute and timbre-editing pairs use natural recordings.Accent-editing pairs share emotion and re-voice both sides with shared timbre; emotion-editing pairs share accent.Source: §A.1 · p. 16
  • Prompts and edit instructions use a shared template inventory generated from seed templates, with approximately 9,000 phrasings and filtering for required placeholders.Template shapes cover all-attribute, timbre-plus-accent, timbre-plus-emotion designs and age, gender, joint age-gender, accent, and emotion edits.Source: §A.1 · pp. 16–17

A.2 Text-Based Evaluation Manifests

The text-based evaluation uses fixed source utterances and controlled prompt or edit manifests, compares StyleStream 2.0 with reference-based and cascaded systems, and specifies objective measures for intelligibility, similarity, and attribute control.

A.2 Text-Based Evaluation Manifests

  • 300 source utterances form both a 3,000-item voice-design set and a 3,000-item voice-editing set, with ten prompts or single-axis edit instructions per source.The source set draws 50 utterances from ESD, 100 from GLOBE, and 150 from LibriTTS; edit targets differ from source labels and instructions name only the edited attribute.Source: §A.2 · p. 17

A.3 Systems and Baseline Generation

  • Reference-conditioned VSC compares StyleStream 2.0 with CosyVoice 2.0 [13], CosyVoice 3.0 [14], Vevo [69], Vevo 2.0 [68], and StyleStream 1.0 [42].Baseline sampling uses recommended NFEs of 32 for Vevo and Vevo 2.0, 10 for CosyVoice variants, and 16 for offline StyleStream 1.0.Source: §A.3 · p. 17
  • Voice-design baselines pair Whisper-large-v3 transcripts with prompts for CapTTS [60] and Qwen3-TTS VoiceDesign [25], while Qwen3-TTS is also evaluated as an offline cascade.The known source transcript is reserved for WER evaluation.Source: §A.3 · pp. 17–18
  • Editing baselines use CapTTS-generated target references with Vevo [69] or Vevo 2.0 [68], routing source and target references according to whether age/gender or accent/emotion changes.For Vevo 2.0, Whisper transcripts guide the output, whose target text remains the source transcript.Source: §A.3 · p. 18

A.4 Objective Evaluation Details

  • VoxProfile [16] classifiers score age/sex, accent, and emotion attributes, with generated audio downmixed to mono, resampled to 16 kHz, and truncated to 15 seconds.Age predictions are collapsed into young and middle-aged categories.Source: §A.4 · p. 18
  • WER uses Whisper-large-v3 transcripts against the known source transcript, while similarity scores compare converted or edited audio with the target or source, respectively.Editing similarity scores are separated into attributes to preserve versus change; higher preservation and lower change scores are better.Source: §A.4 · p. 18

A.5 Subjective Evaluation Protocol

The subjective protocol uses screened English-proficient listeners and five-point ratings across naturalness, style similarity, voice design, and voice editing.

A.5 Subjective Evaluation Protocol

  • 40 Prolific participants are recruited, restricted to first-language English speakers reporting fluent proficiency and at least a 95% approval rating.Submissions completed faster than the expected minimum duration are rejected; listeners are encouraged to use headphones on a laptop or desktop.Source: §A.5 · pp. 18–19
  • Naturalness uses single-sample NMOS ratings, while speaker, accent, and emotion SMOS surveys rate target similarity on a five-point scale.SMOS listeners are instructed to focus on the named style axis and ignore other axes and recording-quality differences.Source: §A.5 · p. 19
  • Voice-design surveys rate naturalness and prompt-style match, while voice-editing surveys rate naturalness, edit match, and preservation of non-edited attributes.Both tasks pair source and test recordings with a natural-language prompt or instruction.Source: §A.5 · p. 19

B Implementation Details

Implementation reuses the StyleStream 1.0 Destylizer and details a Stylizer for speech generation alongside a Voice Controller for text-based design and editing.

B.1 Stylizer

  • The Stylizer is a 16-layer, 768-dimensional DiT that predicts target mel-spectrograms from noisy mel input, content features, and past-mel context.Its output is used to compute average and instantaneous velocity; the past-context channel is zeroed during offline training.Source: §B.1 · p. 19
  • A frozen WavLM-based style encoder pools transformer features into a 768-dimensional style embedding using four perceiver-resampler blocks.The encoder has 97.5M parameters, compared with 128.4M for StyleStream 1.0.Source: §B.1 · p. 19
  • The Stylizer is trained from scratch on Emilia-EN with pixel MeanFlow, then fine-tuned through Self Forcing chunked rollouts while retaining the pMF objective.Fine-tuning uses one ODE step per chunk, randomly sampled chunks of 160–600 ms, and computes loss only on the current chunk region.Source: §B.1 · p. 20

B.2 Voice Controller

  • The Voice Controller is a 16-layer DiT for text-based design and editing, with prompts encoded by frozen Flan-T5 and cross-attention conditioning over style embeddings.Voice design uses a learned null auxiliary embedding; voice editing uses the patchified source style embedding.Source: §B.2 · pp. 20–21
  • The Voice Controller is trained jointly on voice design and editing with an even task split, then samples one target style embedding before streaming and reuses it across chunks.Inference uses a two-pass CFG sampler at NFE 64 and guidance scale ω = 4.Source: §B.2 · p. 21

C Additional Ablation Studies

The ablations examine guidance scale, sampling steps, and text conditioning, identifying operating settings and showing the contribution of pretrained text representations to instruction following.

C.1 CFG Guidance Scale

  • 9.53% WER at ω = 1; for ω ≥ 2, WER remains in a narrow band, and ω = 3 keeps WER below 7.1% within the high-similarity range.The sweep finds shallow similarity optima near ω = 2 for S-SIM and A-SIM and ω = 4 for E-SIM; S-SIM declines as guidance rises.Source: Table 8 · p. 22, §C.1 · p. 22

C.2 Number of Sampling Steps (NFE)

  • 7.06% WER at NFE = 1, with similarity scores within roughly 0.01 of saturated values; all similarity metrics are essentially saturated from NFE = 2 onward.The paper adopts NFE = 4 as a robust operating point, while one- and two-step variants remain options for latency-sensitive deployment.Source: Table 9 · p. 22, §C.2 · p. 22

C.3 Voice Controller Text Conditioning

  • Design gender accuracy rises from 81.17% with trainable token embeddings to 97.61–98.33% with frozen Flan-T5 encoders, while editing gender accuracy rises from 60.00% to 92.17–97.33%.Larger encoders show mixed, attribute-dependent gains; Flan-T5-base performs best on several voice-design metrics, supporting its selection as a balance of semantic conditioning and model size.Source: §C.3 · p. 22

D Limitations

StyleStream 2.0 still has latency and editing-quality gaps, while its broader impacts include potential benefits and misuse risks. The paper also describes responsible-use safeguards and terms governing its external assets.

D Limitations

  • 520 ms latency still leaves room for improvement in tightly interactive use cases.The system reduces end-to-end streaming latency by roughly 2× over StyleStream 1.0.Source: D Limitations · p. 22
  • The timbre interface covers age and gender, while finer-grained qualities such as breathiness, brightness, and vocal effort remain future directions.Emotion editing also trails the corresponding timbre and accent results.Source: D Limitations · p. 22
  • Emotion editing trails the corresponding timbre and accent results, and editing introduces some drift on non-targeted attributes.Further reducing these gaps is left to future work.Source: D Limitations · pp. 22–23

E Broader Impacts

  • Potential applications include accessibility, privacy and anonymization, accent localization, language learning, and creative audio production.The paper also flags risks including voice impersonation, fraud, and synthetic disinformation.Source: E Broader Impacts · p. 23
  • The authors plan to release model weights under a license prohibiting impersonation and malicious applications, with a model card documenting training data, intended uses, and known biases.They support continued research on synthetic-speech detection and audio watermarking.Source: E Broader Impacts · p. 23

F Licenses and Terms for Existing Assets

  • External assets are used for non-commercial academic research, with use restricted to the permitted research and evaluation setting when terms require it.Table 12 summarizes licenses or stated terms for the work's main external datasets, pretrained models, toolkits, and evaluation models.Source: F Licenses and Terms for Existing Assets · p. 23, Table 12 · p. 24

Related papers

Loading 2610.28940v1…