PaperlayerView exact revision on arXiv

Source-linked AI summary

Soloist: A Generative Model for Blind Voice Separation in the Wild

Chen Yang, ChuFan Yu, Botian Jiang, Jingqi Chen, Wenxuan Wang, Yitian Gong, Zhaoye Fei, Xipeng Qiu

arXiv:2610.19716v1cs.SD

TL;DR

Natural recordings rarely include isolated source waveforms, complicating training for voice separation across overlap, noise, and music. Soloist generates voice streams without a target-speaker query or supplied count, combining source and timeline supervision with model-judge reinforcement learning on real recordings. It reports strong results across supplied-count tasks and real recordings with unknown counts, and demonstrates an offline pipeline for continuous speaker tracks and aligned transcripts.

  • Problem

    Natural recordings often have transcripts or speaker timelines but rarely the isolated waveforms needed as references for overlapping-voice separation.

    Source: §1 · p. 1

  • Method

    Soloist autoregressively generates separate audio-codec voice streams and combines constructed-source and speaker-timeline supervision with model-judge reinforcement learning on real recordings.

    Source: §1 · p. 2, §8 · p. 18

  • Results

    With supplied counts, Soloist has the lowest transcription error among compared separators on noisy overlapping speech, speech in noisy environments, and singing vocals; it also separates real multi-speaker recordings with unknown counts well.

    Source: §8 · p. 18

  • Takeaways & Limitations

    An offline pipeline assembles separated outputs into continuous speaker tracks and aligned transcripts from real conversations.

    Source: §1 · p. 3, §8 · p. 19

Abstract

from arXiv · show

Real-world voice separation must recover individual voices from overlapping speech, environmental noise, and music, yet isolated source recordings are rarely available for supervision. We introduce Soloist, a generative model for blind voice separation that autoregressively generates individual voice streams as audio-codec sequences conditioned on the mixture, without requiring a target-speaker query or a predefined voice count. We propose a two-stage training paradigm to learn from synthetic mixtures and adapt to real recordings. The first stage combines supervised separation on synthetic mixtures with isolated source targets and an auxiliary task that predicts speaker timelines from real recordings, learning who speaks when from natural acoustics. The second stage applies reinforcement learning to real recordings, with rewards supplied by a model judge that compares candidate separations based on voice purity and content preservation without isolated source targets. Across diverse scenarios, Soloist recovers clean speech from multi-speaker overlap and noisy recordings, and extracts singing vocals from music accompaniment while maintaining relatively high pitch accuracy and intelligibility. On real-world recordings with unknown voice counts, Soloist also separates voices well. Finally, we integrate Soloist into a full-duplex conversation processing pipeline that assembles separated outputs into continuous speaker tracks, demonstrating its practical utility in real conversations.

1 Introduction

Real-world voice separation must recover speakers and their content from mixtures, but natural recordings rarely provide isolated source waveforms for training. Soloist generates separate voice streams without a target-speaker query or supplied voice count, using multiple forms of supervision and model-judge feedback.

1 Introduction

  • Natural recordings often provide transcripts or speaker timelines but rarely isolated waveforms, while constructed mixtures offer precise targets under constrained acoustics and source combinations.Prior unpaired-mixture methods use mixture-level reconstruction; Soloist focuses on combining several incomplete references with model-generated feedback.Source: §1 · p. 1
  • Soloist autoregressively generates a decoded audio-codec sequence for each voice without requiring a target-speaker query or a supplied voice count.Source: §1 · p. 2
  • Its training recipe combines source-audio targets, teacher-generated song vocals, and speaker timelines, then adapts on real recordings using model-judge rewards.The judge checks candidate reports and order consistency; signal checks filter degenerate outputs and likely copies, while remaining rewards favor voice purity and content coverage.Source: §1 · p. 2, Figure 1 · p. 2
  • The report evaluates Soloist on synthetic, noisy, singing, and real multi-speaker recordings and integrates it into a full-duplex conversation-processing pipeline.Source: §1 · pp. 2–3

2 Related Work

Related work spans supervised and unpaired separation, codec-based generation, diarization, feedback learning, and evaluation. Soloist combines blind variable-stream generation with incomplete-reference supervision and real-mixture feedback, then assembles outputs into continuous speaker tracks.

2.1 Supervised Separation and Variable Source Counts

  • Paired-source separators provide reconstruction baselines but depend on constructed mixtures; permutation handling and recursive or attractor-based methods address source ordering and variable counts [7] [11].Soloist instead represents count, stream boundaries, and voice audio in one serial response, with supplied-count and blind evaluation testing distinct capabilities.Source: §2.1 · p. 3

2.2 Neural Audio Codecs and Generative Separation

  • Discrete codecs support waveform reconstruction and sequence generation, while TokenSplit combines separation, transcription, and generation.Soloist combines variable voice-stream generation, speaker-timeline supervision, and adaptation on real mixtures through candidate feedback.Source: §2.2 · p. 3

2.3 Prompted Extraction and Singing-Voice Separation

  • Prompted extractors use text, audio, visual, or temporal cues [18], whereas blind separation must enumerate voices without a target query; song supervision uses one combined vocal target [19].These target definitions determine what the respective evaluations establish.Source: §2.3 · p. 3

2.4 Learning from Unpaired Recordings and Teacher Targets

  • MixIT and RemixIT learn from mixtures or teacher–student remixes without relying entirely on constructed mixtures with known sources.Soloist instead combines constructed-source audio, teacher-generated song vocals, speaker timelines, and relative model-judge feedback on real mixtures.Source: §2.4 · p. 4

2.5 Speaker Activity, Diarization, and Identity

  • Diarization labels describe speaker activity without specifying waveforms [24], while speaker embeddings associate local outputs with recording-level identities and help detect missing or intruding speech.The pipeline also uses separate content checks because identity agreement does not establish word preservation.Source: §2.5 · p. 4

2.6 AI Feedback, Policy Optimization, and Judge Reliability

  • Prior audio-feedback methods use perceptual, preference, or cross-modal rewards [34]; Soloist compares complete blind multi-voice separations for purity and coverage.Its judge describes clips independently, compares reports against an anonymous mixture anchor, and filters rankings that change under shuffled and reversed candidate orders.Source: §2.6 · p. 4

2.7 Evaluating Content, Acoustics, Pitch, and Identity

  • Evaluation separates reconstruction, recoverable words, pitch, vocal identity, and reference provenance; correct words alone do not establish correct pitch or voice identity.Teacher-extracted vocal references measure agreement with that teacher, unlike isolated-source references.Source: §2.7 · p. 5

2.8 Continuous Separation and Full-Duplex Conversation Data

  • Continuous speech separation must link independently generated pieces to persistent speakers and the original timeline; the pipeline checks purity, missing speech, and content before packaging aligned tracks and transcripts [44].Window planning, stream assignment, alignment, and targeted repair jointly affect coverage.Source: §2.8 · p. 5

3 Generative Blind Voice Separation

Soloist treats blind voice separation as conditional generation of one waveform per inferred voice from a single-channel recording. It serially generates variable numbers of codec-frame streams, with response tokens controlling stream structure and a local decoder producing each frame.

3 Generative Blind Voice Separation

  • Given a single-channel recording, Soloist infers the voice count and generates separate waveforms without a supplied count or target-speaker query, omitting background and residual streams.The response declares a count and emits a variable number of audio-codec streams.Source: §3 · p. 5, §3.1 · p. 5

3.2 Architecture

  • Continuous audio features and discrete mixture codes condition the backbone, which uses a local transformer to generate codec codes within each frame.The architecture uses a fixed codec and trains the audio encoder, backbone, adaptors, and local transformer.Source: Figure 2 · p. 6, §3.2 · p. 6

3.3 Output Format and Inference

  • The text head emits count, stream, and boundary-control tokens; frame requests invoke codec generation until each stream ends, and an end-of-response token terminates output.Training orders target voices by first onset, while evaluation matches generated streams to references by permutation.Source: §3.3 · pp. 6–7

4 Supervised Learning from Multiple Sources

Supervised training uses one model across constructed speech mixtures, song vocals, and natural-recording speaker timelines. The distinct references supervise audio generation or speaker activity, before reinforcement-learning adaptation on real recordings.

4 Supervised Learning from Multiple Sources

  • A single model learns serial voice-stream generation from constructed-mixture sources, song vocal targets, and natural-recording speaker timelines.Audio targets supervise the first two sources; timelines teach who speaks when through timestamp and speaker-label text outputs.Source: §4 · p. 7, Figure 3 · p. 7

4.1 Data Construction

  • Constructed mixtures provide isolated voice targets, diarization intervals provide activity-only text labels, and a fixed separator supplies combined vocal pseudo-targets for songs.Song targets may contain backing vocals or multiple singers, so agreement measures consistency with the teacher rather than recovery of independent studio vocals.Source: §4.1 · p. 8

4.2 Supervised Sampling

  • The sampler groups examples by source count and duration, with timeline rows receiving 4.81% of the sampling budget.Timeline examples form a separate class, and the shortest-duration timeline cell has zero weight.Source: §4.2 · p. 8

4.3 Supervised Objective and Optimization

  • The supervised objective combines teacher-forced text and codec cross-entropies, omits codec loss for timeline rows, and uses no additional count or boundary loss.The codec term is weighted by λ=1.Source: §4.3 · pp. 8–9

5 Reinforcement Learning on Real Recordings

Soloist adapts to real recordings through reinforcement learning (RL) that compares alternative separations without isolated source targets. A model judge supplies relative feedback, while per-stream signals and group-based policy optimization shape updates.

5 Reinforcement Learning on Real Recordings

  • RL adapts the supervised separator on real recordings by sampling alternative separations and comparing voice purity, content coverage, and audio quality without isolated targets.Source: §5 · p. 9

5.1 RL Data

  • The RL prompts use annotated voice counts and cover far-field meetings, film, television, animation, podcasts, short video, and audiobooks.Source: §5.1 · p. 9

5.2 Model-Judge Reward on Real Recordings

  • The judge describes the mixture and candidate tracks independently, then ranks their reports against an anonymous mixture anchor in shuffled and reversed orders.Signal checks screen duration, level, activity, copies, duplicates, and stream-count errors; groups with inconsistent or indistinguishable rankings are excluded from updates.Source: §5.2 · pp. 9–10

5.3 Judge Inference and Failure Handling

  • For long recordings, judge inputs share a uniformly positioned 30-second window; failed descriptions receive floor rewards or can cause the group to be rejected.Source: §5.3 · p. 11

5.4 Per-Stream Feedback

  • Per-track reports provide local feedback for audio-token updates, using audible-speaker counts, artifact penalties, and mostly-silence indicators.The trainer centers stream scores and adds them to corresponding audio-token advantages with coefficient 0.5.Source: §5.4 · p. 11

5.5 Policy Optimization

  • Policy optimization centers judge rewards within candidate groups and uses clipped policy gradients, dynamic filtering, per-stream shaping, KL regularization, and an auxiliary codec loss.Training uses 130 updates without KL regularization, followed by 90 updates with it.Source: §5.5 · pp. 11–12

6 Evaluation

The evaluation covers overlapping speech, noisy speech, and singing, using metrics for content, speaker count, noise, vocal similarity, and pitch. Results report improvements from speaker-timeline supervision, strong judge agreement with PI-WER ordering, and Soloist’s noisy-speech trade-off.

6.1 Evaluation Setup

  • Baselines include speech separators and query-based extractors; supplied-count evaluation sets Soloist’s output count to match the task’s reference count.LibriMix uses two speakers; ViW and singing use one target stream.Source: §6.1 · p. 12

6.1.1 Evaluation Data

  • Evaluation data spans constructed LibriMix mixtures, natural WildTV recordings with one to three speakers, noisy single-speaker ViW clips, and three singing benchmarks.ViW retains twelve conditions covering noise, reverberation, distance, occlusion, and recording-device artifacts; distortion and dropout cases are excluded.Source: §6.1.1 · p. 12

6.2 Metrics and Aggregation

  • The metrics assess transcript preservation and voiced-stream count, background-noise and overall speech quality, plus singing pitch and vocal similarity.DNSMOS BAK and OVRL use a 1–5 scale, with higher scores better; RPA tests melody preservation alongside lyric and vocal-similarity measures.Source: §6.2 · p. 13

6.3 Overlapping Speech

  • The overlapping-speech evaluation includes supplied-count LibriMix mixtures and WildTV recordings, where Soloist and SR-CorrNet infer counts with one model while several baselines use count-specific checkpoints.Table 2 reports LibriMix PI-WER; Table 3 reports WildTV corpus-level PI-WER and blind count accuracy.Source: §6.3 · p. 13, Table 2 · p. 14, Table 3 · p. 14

6.4 Speech in Noisy Environments

  • On ViW, Soloist raises WER from the unprocessed input’s 0.261 to 0.302—the smallest increase among separators—while achieving the highest DNSMOS BAK and OVRL quality.The evaluation tests noise removal alongside preservation of speech intelligibility.Source: §6.4 · p. 14, Table 4 · p. 14

6.5 Singing Vocals

  • Singing-vocal evaluation uses SongYT, MIR-1K, and MUSDB18-HQ, reporting WER, raw pitch accuracy, and SingerSim where references permit.SongYT uses teacher-produced vocal pseudo-references; MIR-1K and MUSDB18-HQ provide isolated vocal references, and MUSDB18-HQ has no lyric references.Source: §6.1.1 · pp. 12–13, §6.5 · p. 15, Table 5 · p. 15

6.6 Training Ablation

  • Speaker-timeline supervision raises SFT count accuracy on every set and lowers PI-WER on noised LibriMix and WildTV, while clean LibriMix PI-WER is nearly unchanged.All rows in the ablation use blind count inference.Source: §6.6 · p. 15, Table 6 · p. 15

6.7 Judge Validation on Known-Source Mixtures

  • The judge agrees with PI-WER ordering in 184 of 207 eligible output pairs (88.9%); the RL model also has lower PI-WER than standard SFT on both LibriMix conditions and WildTV.The known-source test compares pairs whose PI-WER differs by more than 0.2; the judge’s content judgments do not measure voice purity or audio quality beyond transcription effects.Source: §6.7 · p. 16, Table 7 · p. 16

7 Application: Full-Duplex Conversation Processing

The application assembles separated speech into time-aligned speaker tracks and transcripts from diarized conversations. It plans and separates pieces, assigns and repairs streams, checks purity, missing speech, and content, then applies acceptance gates.

7 Application: Full-Duplex Conversation Processing

  • The offline pipeline uses diarized speaker timelines to mine windows and guide separation; embeddings link outputs across pieces, alignment places them on a shared timeline, and failed windows can be re-queried.Accepted outputs are continuous speaker tracks with aligned transcripts.Source: §7 · pp. 16–17, Figure 5 · p. 16

7.1 Window Mining and Planning

  • Window mining requires exactly K speakers, at least 1 s of exclusive speech per speaker, dense speech, and silence-gap boundaries; piece planning covers annotated speech while limiting requested streams to active speakers.For two-speaker windows, overlap strata reflect only intersections present in the speaker timeline.Source: §7.1 · p. 17

7.2 Separation and Assembly

  • Soloist separates each planned piece, then embeddings assign streams to conversation speakers; placement, cross-fading, timeline masks, and verification assemble tracks, while re-queries repair missing or rejected streams.Unrecoverable streams remain explicitly marked as missing.Source: §7.2 · p. 17

7.3 Quality Control

  • Quality control checks intrusive speech, missing speech in exclusive regions, and mismatched content; mismatched utterances are cut out rather than discarding the whole conversation.Regions with too little exclusive speech are marked unassessable and reported separately.Source: §7.3 · p. 18

7.4 Acceptance and Output

  • Acceptance requires a track and transcript for every speaker, passing purity checks, sufficient completeness and least-speaker contribution, no hole-gate failure, and non-silent delivered tracks.Completeness is a coarse automatic proxy that can exceed one; accepted exports include tracks, speaker intervals, utterances, and word timestamps.Source: §7.4 · p. 18

8 Conclusion

Soloist separates voices without a target-speaker query or supplied voice count, combining source-audio and speaker-timeline supervision with reinforcement learning on real recordings. It achieves the lowest transcription error among compared separators on three supplied-count tasks and is also reported to separate real-world recordings with unknown counts.

8 Conclusion

  • Soloist combines source-audio and speaker-timeline supervision with reinforcement learning on real recordings, using a model judge that compares track descriptions under repeated candidate orderings.It separates individual voices without a target-speaker query or supplied voice count.Source: §8 · p. 18
  • Soloist achieves the lowest transcription error among compared separators on noisy overlapping speech, speech in noisy environments, and singing vocals when voice counts are supplied.The singing results also show that lyric content, pitch accuracy, and vocal similarity capture different aspects of separation quality.Source: §8 · p. 18
  • The model is integrated into an offline pipeline that assembles continuous speaker tracks and aligned transcripts from real conversations, alongside a recipe for learning from constructed mixtures and natural recordings with incomplete references.The paper also reports good separation on real-world multi-speaker recordings with unknown voice counts.Source: §8 · pp. 18–19

A.1 Model Configuration

Table 8 presents the model configuration.

A.1 Model Configuration

  • Table 8 lists the model configuration.The supplied passage does not specify the configuration values.Source: Table 8 · p. 23

A.2 Optimization Settings

The optimization uses a clipped token-level policy surrogate within a loss that also includes KL regularization and an auxiliary codec cross-entropy term. The listed RL settings distinguish the two phases by learning rate and KL coefficient.

A.2 Optimization Settings

  • The clipped surrogate uses each token’s likelihood ratio against the old policy and its advantage; the policy loss averages the negative surrogate over eligible tokens.The loss also includes a KL penalty and codec cross-entropy auxiliary term, while audio likelihood ratios use truncated importance sampling.Source: §A.2 · p. 23
  • Only the learning rate and KL coefficient change between phase 1 and phase 2 in the RL configuration.The enabled KL coefficient is 0.001 with the k1 estimator, the auxiliary coefficient is 0.05, and centered stream feedback has weight 0.5; phase 1 uses a zero KL coefficient.Source: Table 9 · p. 23, §A.2 · p. 23
  • The RL group size is 8, and groups are filtered using a spread threshold of 0.005.Centered per-stream feedback augments audio-token advantages.Source: §A.2 · p. 23

A.3 Output Interface and Exact Prompts

The appendix specifies Soloist’s output interface, training targets, reward-judge procedure, and pipeline settings. Its prompts structure separation and timeline outputs, while the judge compares candidate-track reports and signal checks shape reward eligibility.

A.3 Output Interface and Exact Prompts

  • At input-frame positions, the model sums projected continuous features with embedded mixture codes; at output-frame requests, it generates twelve codes that the output adaptor embeds for subsequent steps.The interface reuses the Qwen3 vocabulary and chat template, assigning audio roles to existing tokens.Source: §A.3 · p. 24, Table 10 · p. 24
  • Teacher forcing supervises assistant text-channel positions, while audio-code cross-entropy applies only at output-frame positions; separation targets follow first onset and timeline prediction uses text alone.The separation prompt requests a speaker count followed by indexed audio, whereas the timeline prompt requests sorted speaking intervals without speech content.Source: §A.3 · p. 24

B.1 Inference Settings

  • Judge inference settings are listed in Table 11, and tier scores are averaged over two rankings.The supplied passage does not provide the setting values.Source: Table 11 · p. 25, §B.1 · p. 25

B.2 Signal Gates and Reward Shaping

  • Signal gates operate at 16 kHz; the count penalty uses the number of streams passing duration, level, and activity tests, and copy checks allow codec-reconstruction tolerance.A mixture-like output is exempted from copy checks for a declared single source when the count annotation is at most one.Source: §B.2 · p. 25, Table 12 · p. 25
  • Dead candidates receive severity tiers for passthrough, missing valid streams, overlong output, or zero count penalty, while rejected groups receive zero reward.Reward mapping also applies copy/duplicate, duration, and count penalties; the alive guard was inactive for surviving candidates in recorded runs.Source: §B.2 · pp. 25–26, Table 13 · p. 26

B.3 Exact Judge Prompts

  • The judge ranks candidate separations from independent reports of the mixture and isolated tracks, prioritizing speaker separation, content coverage, duplicate avoidance, and artifacts.Candidates are ranked into quality tiers, and the prompt allows ties while asking for multiple tiers when reports differ materially.Source: §B.3 · pp. 26–27

C Full-Duplex Pipeline Parameters

Related papers

Loading 2610.19716v1…