PaperlayerView exact revision on arXiv

Source-linked AI summary

The Voice on the Line

Husein Zolkepli

arXiv:2610.35272v1cs.SDcs.CL

TL;DR

A transcriber using one microphone also captures nearby voices, and short evaluation windows miss failures involving long turns and lone callers. This paper trains causal voice isolators and evaluates them on longer calls and real crosstalk; WavLM-Large improves both failure modes, while a smaller student is designed for CPU use.

  • Problem

    A one-microphone transcriber also hears nearby speakers, and short evaluation windows do not test forgetting the caller during long background turns or damaging a lone caller.

    Source: §1 · p. 1, §11 · p. 36

  • Method

    The paper trains causal isolators that enroll the caller from the call itself, evaluates them on long meeting windows and real call-centre calls, and distills the release into a CPU student.

    Source: §1 · p. 2, §9 · pp. 30–31

  • Results

    WavLM-Large’s causal weights improve both long-turn forgetting and lone-caller damage, while a student imitating WavLM, w2v-BERT 2.0, and Whisper’s encoder do not provide the same improvement.

    Source: §11 · p. 36

  • Takeaways & Limitations

    The release and a 22.1M-parameter CPU student are open with a LiveKit plugin.

    Source: §1 · p. 2, §9 · p. 30

    The live enrollment rule can select the wrong voice when someone else is loudest, and recogniser scores may differ with other recognisers.

    Source: §12 · p. 36

Abstract

from arXiv · show

A transcriber listening through one microphone also hears everyone near it. We train causal voice isolators: masking transformers of 140M, 243M and 374M parameters, 20 ms of latency, enrolling the caller from the call itself. Short evaluation windows hide two failures that real calls expose: forgetting the caller when another voice talks alone for a while, and damaging a caller who is alone. Long calls with turns, memory beyond the attention window, real headset crosstalk and drifting caller levels address both, but each trades one against the other. WavLM-Large's own weights, made causal and trained with the model, move the trade: M with them (555M parameters) lowers the error on our development sets from 11.44% to 10.55%; a student trained from scratch to imitate WavLM does not, and w2v-BERT 2.0 and Whisper's encoder, pre-trained on far more audio, help less or not at all. A lone caller loses most at the start of a recording: no training call opened with someone already talking. Training on windows cut from longer calls lowers the error on clean development calls from 3.92% to 2.96%; this is our release. Live, the release makes 16.13% word errors on 60 s AMI windows, against 23.66% for VoxLock, the strongest open isolator, and 10.19% on its own AMI windows, against 11.71%; it is level on Krisp's benchmark (5.74% against 5.63%), and a caller who is alone still loses more words than with VoxLock (5.58% against 2.92% on clean test calls). Distilled into 22.1M parameters without WavLM, it runs live on one CPU thread inside a LiveKit agent, at 33% of a core per call in 20 ms frames, with 10.66% error on our development sets (the release 10.49%). We open-source the release and this CPU model, with a LiveKit plugin, at https://huggingface.co/collections/Scicom-intl/voice-isolator.

1 Introduction

A single microphone captures both the caller and nearby speakers, while ordinary noise suppression keeps all speech. The paper trains causal isolators that enroll the caller from the call itself and evaluates them against failures short windows miss.

1 Introduction

  • 23.19% of words are wrong for Parakeet TDT 0.6B v2 on Krisp’s 265 real recordings with a second voice, almost all other people’s words.A noise suppressor keeps those voices because they are speech; VoxLock is the strongest open isolator described here.Source: §1 · p. 1
  • Krisp’s comparison plots word error rate against leaked words per minute of background-only speech, with one point per system.The plot distinguishes suppressors, which keep other voices, from isolators, which remove them.Source: §1 · p. 2
  • The paper trains three sizes of causal isolator, enrolling from the loudest 3 s heard so far with no look-ahead beyond a 20 ms window.The contributions include long meeting windows, real call-centre calls, training stages for two failures, a GPU server, a CPU student, and a LiveKit plugin.Source: §1 · p. 2

2 Background

The background distinguishes removing non-speech from selecting or isolating a particular voice. It also motivates evaluation on real recordings and with latency constraints relevant to live use.

2 Background

  • Noise suppression removes everything that is not speech, whereas target speaker extraction keeps one voice given a clue such as an enrollment clip.Primary-speaker isolation instead keeps whoever is on the microphone; nearness alone is a weak cue, motivating VoxLock’s enrollment from the call.Source: §2 · p. 2
  • Simulated mixtures can overstate extraction on real conversations, while Krisp, AMI, and NOTSOFAR provide real recordings or meeting channels with crosstalk.Krisp labels who talks when; AMI and NOTSOFAR use head-worn microphones that also hear other participants.Source: §2 · pp. 2–3
  • The 2023 DNS challenge caps STFT window plus look-ahead latency at 20 ms; VoxLock is 24 ms and DeepFilterNet3 is 40 ms.For GPU serving, a call also waits for its chunk and the batched step.Source: §2 · p. 3

3 Data

The data combine simulated calls with long-call training examples and real crosstalk from held-out meetings. Evaluation sets include long background-only stretches and call-centre recordings to test both leakage and damage to a lone caller.

3 Data

  • Training examples simulate calls from clean speech, with the target being the microphone-side speaker alone through the mixture’s room position, device, and band.The target therefore does not require invented bandwidth or retaining other speakers.Source: §3 · p. 3

3.1 SPEECH, NOISE AND ROOMS

  • The speech pool contains 2,761 h across 131 Common Voice languages and more, with speakers split 199 to 1 between training and development.The data also include 51.3 h of noise from five corpora and 20,000 simulated rooms, with the last 1,000 held out.Source: §3.1 · pp. 3–4

3.2 SYNTHETIC CROSSTALK, NOISE AND THE LINE

  • Synthetic calls add up to three background talkers, mostly during caller pauses, and up to two noises before applying the device and line.Seven examples in ten also include 2–5 s of noisy or clean enrollment from the same voice; the rest teach primary-speaker finding without enrollment.Source: §3.2 · pp. 5, 7

3.3 LONG CALLS

  • Long-call training uses 24 s calls where the caller returns after a background voice speaks alone, creating 4.5 s per call of such stretches over 8 s.Six-second examples never have caller silence longer than 3 s, so they do not teach this memory challenge.Source: §3.3 · p. 7

3.4 REAL CALLERS

  • Real-caller training varies mixture level, caller-level drift, and whether anyone else is present; three examples in ten are cut from an already ongoing call.Mixture levels span −65 to −10 dBFS, and half the long calls have caller-level gains drawn every 1 to 4 s.Source: §3.4 · p. 8

3.5 REAL CROSSTALK

  • Real crosstalk comes from 362 held-out AMI, NOTSOFAR, and AliMeeting meetings totaling 620 h.A wearer’s channel is used as an exact target when nobody else speaks and as natural-level crosstalk when only others speak.Source: §3.5 · p. 8

3.6 EVALUATION SETS

  • Evaluation adds 60 s AMI windows with at least 8 s of others-only speech, plus call-centre recordings scored with and without another call’s speaker added.Krisp-style windows stop at 4 s of background-only speech; the additional call-centre evaluation measures damage to a lone caller against plain recognition.Source: §3.6 · p. 8

4 Model

The paper compares a causal masking transformer with an offline restoration architecture. Its real-time model conditions on an enrolled speaker, and training combines signal, spectral, and silence-energy objectives in stages.

4 Model

  • The realtime model is a causal masking transformer, while the comparison architecture is Sidon’s restoration model at 16 kHz.The realtime model is built for GPU serving in three sizes.Source: §4 · p. 9

4.1 REALTIME: A CAUSAL TRANSFORMER OVER FRAMES

  • The realtime transformer processes 20 ms frames using band encoding and temporal LSTMs, attends over the last 10 s, and predicts a complex mask.The release adds causal WavLM-Large layers mixed into its tokens; the model sizes differ only in width.Source: §4.1 · p. 9

4.2 ENROLLMENT AND SELF-ENROLLMENT

  • A speaker vector enters every layer through FiLM, using either an enrollment clip or a learned null vector.Live calls start with the null vector, then enroll the loudest 3 s heard from 3 s onward, fixing that choice once the call is 15 s old.Source: §4.2 · p. 10

4.3 OFFLINE: SIDON’S ARCHITECTURE AT 16 KHZ

  • The offline Sidon-architecture model predicts caller-alone w2v-BERT 2.0 features and resynthesises them with a DAC decoder.The two model sizes are trained on the paper’s calls and output at 16 kHz.Source: §4.3 · p. 10

4.4 TRAINING

  • Training combines negative signal-to-noise ratio, multiresolution spectral distance, and output energy during caller silence across three stages.The WavLM branch joins at the second stage, and training choices are made on development sets.Source: §4.4 · p. 11

5 Evaluation

The evaluation runs open systems on shared 16 kHz files and tests the models in live and alternative enrollment modes. Scoring uses different recognizers and normalization procedures across call and meeting sets.

5 Evaluation

  • Open systems are run on the same 16 kHz files, with VoxLock tested under its published protocol and live mode, alongside DeepFilterNet3 and Krisp VI 2.5.Our models are tested live, with VoxLock’s look-ahead sample, without enrollment, or with an oracle clip.Source: §5 · p. 11
  • Krisp’s published outputs are evaluated alongside our models, while VoxLock’s published-protocol and live results provide two comparison settings.The models are also assessed with no enrollment or an oracle clip.Source: §5 · p. 11
  • Krisp and ai-coustics calls use Parakeet TDT 0.6B v2 with Krisp normalization; meetings use Parakeet v3, and AliMeeting uses Whisper large-v3.Intervals are computed from 2,000 resamples of callers.Source: §5 · p. 12

6 Results

The results show how the release compares with existing systems and how training, enrollment, and model choices affect leakage and caller preservation. Its gains on calls with background speech coexist with a remaining weakness on clean calls where the caller is alone.

6.1 REPRODUCING THE BASELINES

  • The evaluation reproduces VoxLock’s Krisp results: 262 of 265 files have identical error counts, and published Krisp outputs reproduce VoxLock’s rows exactly.VoxLock’s PyTorch script scores 4.99% on Krisp and 10.26% on its AMI windows; its streaming ONNX package scores 5.19% and 11.26%.Source: §6.1 · p. 12

6.2 AGAINST EXISTING SYSTEMS

  • On test sets with a second voice, the release makes fewer errors on every meeting and call set, is level with VoxLock on Krisp, and has 5.58% against 2.92% on clean calls.The clean-call comparison is a change against plain recognition; the meeting and call comparison is release live against VoxLock live.Source: §6.2 · p. 12
  • Windowed models forget the caller after another voice talks alone, while the long-call stage trades better suppression for greater damage to a lone caller.The window cut reduces the caller’s first-3-second loss on test calls from 23% to 7%; the release still has 5.58% clean-call error against VoxLock live’s 2.92%.Source: §6.3 · pp. 12, 13, 17

6.5 ABLATIONS

  • Synthetic crosstalk, noise, and rooms matter most in stage 1; without the enrollment branch, short windows leak 19.03 words per minute on Krisp long calls, against 5.88.English-only speech, omitting restored speech, and real crosstalk each help; the last two enter the release recipe.Source: §6.5 · pp. 18–19

6.6 ENROLLMENT

  • The loudest 3 seconds heard so far outperform enrollment from the model’s own output; omitting enrollment is kinder to a lone caller but worse on long meetings.Only live enrollment modes can run in a call.Source: §6.6 · p. 19

6.7 MODEL SIZE

  • M outperforms S, while L does not outperform M; with WavLM, their dev-4 errors are 10.81%, 10.55%, and 10.66%, respectively.Without WavLM, the corresponding errors are 11.67%, 11.44%, and 12.04%; these results precede the window cut, which only the release uses.Source: §6.7 · pp. 19–20

6.8 SELF-SUPERVISED SPEECH MODELS

  • Training WavLM’s own causal weights with the model lowers dev-4 from 11.44% to 10.55%, while w2v-BERT 2.0 and Whisper’s encoder reach 11.35% and 11.74%.A scratch student imitating WavLM does not improve dev-4; HuBERT variants and larger alternatives also do not match WavLM’s result.Source: §6.8 · pp. 20–21

6.9 MASKING OR RESYNTHESIS

  • Sidon-architecture models resynthesizing self-supervised features keep every voice, because features of a mixture carry all speakers.The models are trained on the same calls as ours.Source: §6.9 · p. 22

7 Serving

The GPU serving design maintains per-call state and runs one network step every 20 ms. Stress tests measure concurrent-call capacity, end-to-end latency, and optimizations to the serving kernels.

7 Serving

  • Each call keeps its attention cache, LSTM states, and speaker vector in a state bank; the GPU runs a batched network step every 20 ms.The release’s WavLM branch maintains a separate state bank, and its step agrees with the offline release to 69 dB.Source: §7 · pp. 23–24

7.1 STRESS TESTS

  • With 20 ms chunks, one GPU serves 512 calls with S, 384 with M, and 160 with the release; the release uses 47.7 GMAC per second of audio against 28.5 for M alone.The step-time test defines real-time capacity as staying below the dashed line.Source: §7.1 · p. 24
  • End to end over WebSockets, the release holds 128 calls at 20 ms chunks with 27 ms median and 38 ms 99th-percentile round-trip delay when connections are spread across eight processes.With one front-end process, it holds 32 calls at 21 ms median and 33 ms 99th-percentile delay; M keeps up with all 256 calls tried across eight processes.Source: §7.1 · pp. 24–25
  • Custom attention kernels and PyTorch LSTMs reduce a 64-call step from 18.8 ms to 11.6 ms, with output agreement of 69 dB.Over 30 minutes at 64 release calls, no calls were lost and the 99th-percentile delay remained 32 ms.Source: §7.1 · pp. 25–27

8 In a voice agent

The LiveKit integration carries call-level state across incoming frames and places the isolator before the VAD, turn detector, and recogniser. On recordings, it reduces background-voice speech detections and shortens delayed turn endings, while sometimes missing caller speech.

8 In a voice agent

  • The isolator runs as a LiveKit frame processor ahead of the VAD, turn detector, and recogniser, with per-call state and a shared ONNX Runtime session.The exported stream uses numpy for STFT, overlap-add, attention rings, and live enrollment, and reproduces PyTorch output to 124 dB with live enrollment.Source: §8 · p. 27
  • LiveKit frames do not truncate context: the caller vector uses the loudest 3 s heard, attention covers 10 s, and two LSTMs retain whole-call history.Before 3 s, no speaker vector exists, and the model can remove some initial words from a lone caller.Source: §8 · p. 28
  • Background-only speech starts fall to 0.13, 1.77, and 1.71 per minute behind the release on Krisp, ai-coustics, and AMI, from 5.09, 9.03, and 2.53 on raw calls.Behind GTCRN, the corresponding counts are 6.54, 10.54, and 3.91 per minute.Source: §8 · p. 28
  • 47.9% of caller pauses contain VAD-detected speech on raw calls, versus 22.8% behind the release; turn-end p90 falls from 2.43 s to 1.00 s, and missed turn ends from 28% to 18%.On lone-caller speech, the release VAD misses 5.2%, compared with 2.6% for VoxLock and 0.6% for GTCRN; most release misses occur in the first seconds.Source: §8 · p. 29
  • The CPU student retains much of the VAD gain: speech is detected in 26.2% of caller pauses, turn ends are seen 1.34 s late at p90, and 4.9% of lone-caller speech is missed.Source: §8 · p. 30

9 On one CPU core

The distilled CPU student runs through the LiveKit plugin on one thread, while its live word-error results vary across test sets. Distillation from the release transfers much of its clean-caller performance without retaining WavLM.

9 On one CPU core

  • The student keeps the release’s band encoder, decoder, mask head, and enrollment encoder, shrinks the transformer, drops WavLM, and learns from both release output and clean-caller targets.Source: §9 · p. 31
  • 10.66% dev-4 error for teacher-trained C22 approaches the release’s 10.49%, versus 12.14% for C22 trained on the recipe alone.C22 reaches 2.84% on clean dev calls when taught by the release, versus 7.00% from scratch and 2.96% for the release.Source: §9 · p. 31
  • In 20 ms frames, the student uses 33% of one core per call, compared with 132% for VoxLock’s default model and 5% for GTCRN.C22 is selected as the student: its dev-4 error is 10.66%, versus 11.06% for C10 and 10.78% for C51.Source: §9 · p. 32
  • 8-bit quantisation uses 23% of a core in 20 ms frames and yields 10.52% dev-4 error, versus 10.66% in float, but outputs depend on frame size.The same call in 20 ms and 50 ms frames agrees to 40 dB in 8-bit mode, versus 104 dB in float.Source: §9 · p. 33
  • 15.55% against 23.66% on 60 s AMI windows and 10.29% against 12.07% on AMI test meetings favor the student over VoxLock live.The student also scores 7.58% against 8.84% on NOTSOFAR and 49.90% against 56.65% on calls with another voice.Source: §9 · p. 33
  • On Krisp, the student is level with VoxLock at 5.66% and 5.63%; on clean test calls with a lone caller, it trails VoxLock at 5.32% against 2.92%.The release also trails VoxLock on lone-caller clean test calls, at 5.58% against 2.92%.Source: §9 · p. 34

10 In a running agent

Tests inside running LiveKit agents show that the isolator runs with bounded delay and reduces VAD responses to background voices. The CPU and GPU models retain the recorded-call VAD pattern, while still missing some caller speech.

10 In a running agent

  • The real-agent test streams 32 calls with another voice simultaneously, compares against each caller’s own channel, and measures both isolator cost and VAD behavior.The final timing-table row uses 64 calls.Source: §10 · p. 34
  • On the CPU, the student takes 15.21 ms per 50 ms frame at the median and 24.88 ms at p99, with 10 ms output delay.On the GPU, the caller returns 40 ms late; audio exceeding the 40 ms cap goes through unfiltered, affecting 0.1% of audio at 32 calls and 1.0% at 64.Source: §10 · p. 35
  • Behind the GPU and CPU models, the VAD detects speech in 24.9% and 27.7% of caller pauses, versus 47.5% without filtering; turn-end p90 is 1.06 and 1.03 s, versus 2.04 s.The VAD misses 6.9% and 3.0% of caller speech behind the GPU and CPU models, respectively.Source: §10 · p. 35
  • The agent’s turn detector can answer after the caller stops, but raw background speech keeps the VAD waiting; isolation leaves only the caller, with delay added to the turn end.The described isolator delay is 10 ms on CPU and 40 ms through the GPU server.Source: §10 · p. 35
  • The deployment fixes cap late-server delay and pass later audio unfiltered; a stream opening in silence now waits to enroll until caller audio is available.GPU serving was also changed to step within its tick budget rather than again for every incoming frame.Source: §10 · pp. 35–36

11 Conclusion

The conclusion frames voice isolation as balancing removal of other speakers against preserving a lone caller. It reports gains over VoxLock in several live settings, continued weakness on lone-caller speech, and a CPU student and open release.

11 Conclusion

  • The release beats VoxLock live on long meetings and calls with another voice, is ahead or level on other test meetings and Krisp, but remains behind on lone-caller calls.Source: §11 · p. 36
  • Longer-call memory and real-call conditions expose a trade-off: the model can forget the caller during long background-only speech or damage a caller who is alone.Source: §11 · p. 36
  • Causal WavLM-Large weights move that trade-off, whereas a student imitating WavLM, w2v-BERT 2.0, and Whisper’s encoder help less or not at all.Source: §11 · p. 36
  • Cutting training windows from longer calls reduces cold-start losses because the original training calls did not open with someone already talking.Source: §11 · p. 36
  • The release costs about 47.7 GMAC per second of audio, and one GPU carries 160 live calls; the release and CPU student are open with a LiveKit plugin.Source: §11 · p. 36

12 Limitations

The limitations section describes evaluation and selection constraints, alongside the trade-offs explored in the lone-caller and stage-1 ablations. Its tables also document the tested recipe variants and comparison conventions.

12 Limitations

  • Evaluation uses scripted Krisp calls and call-centre windows with a mixed-in second voice; recognisers and selection choices also constrain the reported results.Results depend on Parakeet or Whisper transcripts, and different weighting can select a different model.Source: §12 · p. 36

A Tables

  • Appendix tables cover full test results, recipe and stage ablations, model sizes, speech encoders, and enrollment modes; bold marks the best live system per column.VoxLock’s benchmark enrollment and PyTorch script look ahead up to 15 seconds.Source: A Tables · p. 41

A lone caller

  • The lone-caller variants change silence gating, enrollment, and window cutting, with the release using windows cut from longer calls.The reported rows also include a no-enrollment variant and a condition with an initial second of digital silence.Source: A lone caller · p. 42

TABLE 14

  • The stage-1 ablation includes a keep-term hinge for turning the caller down, tested at weight 2.The table describes simulated-set SI-SDR improvement and real-set word error rate and leaked words per minute.Source: TABLE 14 · p. 44

B Full results

On 60-second AMI windows, the live M + WavLM system reports a 16.13% word error rate and a lower error than VoxLock’s PyTorch script. On AMI’s 134 VoxLock windows it is close to that script, while the simulator parameters specify varied speech and channel conditions.

B Full results

  • 10.19% word error rate for live M + WavLM on AMI’s 134 VoxLock windows, with a −0.07-point difference to VoxLock’s PyTorch script.The 95% interval for the difference is [−0.41, +0.40].Source: B Full results · p. 45
  • 10.18% word error rate for live M + WavLM on AMI evaluation meetings, with a −0.13-point difference to VoxLock’s PyTorch script.The 95% interval for the difference is [−0.97, +0.52].Source: B Full results · pp. 45–46
  • 16.13% word error rate for live M + WavLM on 60 s AMI windows, with a −4.07-point difference to VoxLock’s PyTorch script.The reported 95% interval for the difference is [−7.02, −1.68].Source: B Full results · pp. 45–46

C Simulator parameters

  • The simulator draws varied conditions, including a 10% chance of a late primary start and an 8% chance of a real codec on the mixture.Its defaults also include talkers placed in primary pauses and codecs held out for testing.Source: C Simulator parameters · p. 48
Loading 2610.35272v1…