RadarVox: Radar-Audio Multimodal Cocktail-Party Speech Separation with Speaker-Aware Cross-Modal Matching

I. INTRODUCTION

• Audio-only blind source separation estimates source-specific waveforms but leaves outputs unordered and does not identify their physical sources.The introduction frames the task as jointly recovering speech and assigning each output to a source.Source: §I · p. 1
• No publicly available radar-audio dataset was designed for identity-aware cocktail-party separation with explicit source correspondence.Existing audio-only datasets lack auxiliary physical observations, while radar datasets have limitations including single-source protocols.Source: §I · p. 2
• RadarVox pairs acoustic mixtures with source-level radar displacement sequences across loudspeaker-emitted and natural human speech settings.The human subset uses TIMIT recordings from 19 speakers to generate clean and noisy two- and three-speaker mixtures.Source: §I · p. 2
• The proposed two-stage framework injects radar motion priors into a DPRNN separator and matches unordered speech streams to radar-observed speakers.The paper presents this as a way to mitigate the permutation ambiguity of acoustic separation and support explicit source assignment.Source: §I · p. 2

A. Audio-Only Speech Separation

• Audio-only methods use source-discriminative embeddings, permutation-invariant training, or waveform-level encoder-separator-decoder models, including DPRNN.Heavy spectral overlap, similar speakers, noise, and reverberation can weaken acoustic distinguishability.Source: A. Audio-Only Speech Separation · p. 3

B. Radar-Only Speech Sensing

• Radar-only speech recovery is limited because mechanical motion lacks parts of the radiated acoustic waveform and practical mmWave systems have sampling-rate and bandwidth limits.The paper therefore positions radar as an auxiliary source of speech-related motion cues rather than a standalone replacement for audio.Source: B. Radar-Only Speech Sensing · p. 3

C. Radar-Audio Speech Enhancement and Separation

• Radar-audio fusion has mainly served single-speaker enhancement, while RadioSES extends it to multi-speaker separation without associating outputs with physical sources.The section characterizes explicit output-source correspondence in cocktail-party separation as unexplored.Source: C. Radar-Audio Speech Enhancement and… · p. 3

III. RADARVOX BENCHMARK FOR IDENTITY-AWARE SPEECH SEPARATION

• RadarVox supports identity-aware cocktail-party separation, single-speaker enhancement, and extended modality analysis of radar-derived speech-related motion cues.The benchmark section describes recording scenarios, radar acquisition and preprocessing, and dataset composition.Source: §III · p. 3

1) Loudspeaker-Emitted Speech:

• Loudspeaker diaphragm vibration provides a stronger, more stable controlled testbed for radar parameter selection and displacement-extraction validation.The setting is intended to validate the sensing pipeline before human speech recording.Source: 1) Loudspeaker-Emitted Speech · p. 3

2) Single-Speaker Human Speech:

• Human speech sensing localizes the laryngeal region, but radar motion is weaker and less mechanically constrained than loudspeaker vibration.The microphone remains the primary acoustic observation, with radar serving as an auxiliary physical modality.Source: 2) Single-Speaker Human Speech · p. 4

3) Multi-Speaker Human Speech:

• The main separation benchmark generates two- and three-speaker mixtures from single-speaker human recordings following the LibriMix protocol.Additional real overlapped recordings use speakers seated at different distances from the radar to assess practical feasibility.Source: 3) Multi-Speaker Human Speech · p. 4

1) Parameter Design:

• Approximately 11 kHz effective chirp-time sampling is obtained by preserving all in-frame chirps rather than retaining one chirp per frame.The paper notes that prior systems’ 1–2 kHz slow-time sampling limits recoverable vibration bandwidth, and speech intelligibility falls below 50% when band-limited to 2 kHz.Source: 1) Parameter Design · p. 4

2) Preprocessing Choices:

• The preprocessing pipeline transforms each chirp into a range profile, uses noncoherent accumulation to select a speech-related scatterer, then applies phase-aligned coherent accumulation in sliding windows.PACA locally aligns phase and averages coherently to improve temporal consistency without assuming global phase stationarity.Source: 2) Preprocessing Choices · pp. 4–5
• PACA achieves the best SI-SDR among the evaluated loudspeaker radar preprocessing strategies.The comparison also evaluates EMA clutter cancellation, VMD, wavelet shrinkage, and Kalman smoothing.Source: 2) Preprocessing Choices · p. 5, Table III · p. 5

• The section focuses on high effective chirp-time sampling and range-based source separability as radar capabilities for speech separation.Source: C. Radar-Sensing Capabilities for… · p. 5

1) Chirp-Time Sampling rate:

• At 11087 Hz, radar-derived speech representations have lower WER and stronger SI-SDR than lower-rate settings.The paper links dense chirp-time sampling with preserving speech-related motion cues.Source: 1) Chirp-Time Sampling rate · p. 5

2) Range-Based Source Separability:

• A single FMCW radar separates motion signatures by source distance, providing geometric constraints for matching separated streams to physical sources.The configured range resolution is 4.30 cm.Source: 2) Range-Based Source Separability · p. 5

D. Dataset Corpus and Split Protocol

• RadarVox combines a loudspeaker corpus for embodied agent–agent interactions with a TIMIT human–agent speech corpus of 19 speakers, split speaker-disjointly into 11 training, 4 validation, and 4 test speakers.Multi-speaker mixtures are generated independently within each split to preserve speaker disjointness.Source: D. Dataset Corpus and Split Protocol · pp. 5–6

IV. METHODOLOGY

• Given an acoustic mixture and source-level radar displacement sequences, the two-stage framework estimates separated speech streams and assigns each to its corresponding radar-observed source.Source: §IV · p. 6

A. Human-Speech Displacement Refinement

• Human-speech displacement refinement suppresses large non-phonatory motion artifacts and interpolates corrupted segments before bandpass filtering.The displacement threshold is 200 µm, set based on empirical validation and reported micrometer-scale neck-surface displacement.Source: A. Human-Speech Displacement Refinement · pp. 6–7

1) Audio encoder:

• The audio encoder maps the mixture through a convolutional front end and two DPRNN blocks to produce a 256-channel acoustic representation.The DPRNN blocks alternate intra-segment and inter-segment recurrent modeling.Source: 1) Audio encoder · p. 7

2) Radar encoder:

• The radar encoder processes each source’s displacement sequence with convolutional and DPRNN layers, then temporally interpolates its representation to align with the acoustic representation.Its radar representation has 64 channels.Source: 2) Radar encoder · p. 7

3) Radar-Conditioned Cross-Modal Fusion:

• Radar features from all speakers are projected into the acoustic latent space and injected through a gated residual connection.The radar residual is produced by a zero-initialized pointwise convolution; α is the learnable fusion gate.Source: 3) Radar-Conditioned Cross-Modal Fusion · p. 7

4) Mask Estimation and Decoding:

• A four-layer DPRNN predicts soft masks for each output stream, which are applied to the latent mixture features before time-domain decoding.The decoder uses linear basis transformation and overlap-and-add synthesis.Source: 4) Mask Estimation and Decoding · p. 7

C. Speaker-Aware Audio-Radar Cross-Modal Matching

• The matcher maps separated-audio features and source-level radar displacement into normalized global embeddings whose inner products form a cross-modal similarity matrix.Its radar encoder shares a convolutional and DPRNN backbone with the fusion radar encoder, but produces global identity embeddings.Source: C. Speaker-Aware Audio-Radar… · p. 7
• A bidirectional matching loss uses the separator’s PIT-optimal assignment as a pseudo target to learn stream-source alignment while the separator remains frozen.This trains cross-modal correspondence without changing the separator’s separation capability.Source: C. Speaker-Aware Audio-Radar… · p. 8

1) Stage I Front-end Fusion Separator:

• The Stage-I separator is trained with negative SI-SDR for single-speaker enhancement and permutation-invariant training for two- and three-speaker separation.PIT minimizes over all permutations of the K sources.Source: 1) Stage I Front-end Fusion Separator · p. 8

2) Stage II Speaker-Aware Cross-Modal Matcher:

• Stage II trains a speaker-aware matcher to associate separated streams with radar identities using a bidirectional matching loss.The target permutation is inferred from pseudo labels.Source: 2) Stage II Speaker-Aware Cross-Modal… · p. 8

A. Evaluation Protocol and Metrics

• Evaluation reports SI-SDR, SIR, STOI, and PESQ, with both PIT and ordered metrics for multi-speaker separation and matching accuracy for source assignment.WER and CER serve as reference-free feasibility indicators for real-world recordings without clean references.Source: A. Evaluation Protocol and Metrics · p. 8

B. Single-Speaker Speech Recovery and Enhancement

• The single-speaker protocol treats radar-only reconstruction as speech recovery and audio-only or audio-radar processing as enhancement of noisy input audio.Table V presents performance for these sensing modalities and fusion settings.Source: B. Single-Speaker Speech Recovery and… · p. 8, Table V · p. 8

1) Audio-only baselines:

• Conv-TasNet and DPRNN substantially improve intelligibility and perceptual quality over noisy input in the single-speaker setting.The noisy input remains a direct acoustic observation of target speech, enabling audio-only models to exploit its acoustic structure.Source: 1) Audio-only baselines · p. 8

2) Radio-only baselines:

• WaveEar improves SI-SDR by 9.34 dB over raw radar input, but radar-only methods remain far from intelligible or high-fidelity speech recovery.The authors explain that radar captures only a partial mechanical correlate of speech, making it more suitable as an auxiliary cue than a standalone reconstruction solution.Source: 2) Radio-only baselines · p. 8

3) Audio and radar fusion methods:

• RadioSES achieves the highest SI-SDR of 12.07 dB among the described audio-radar fusion methods, while RadarVox achieves comparable performance.The authors attribute the comparison to radar serving as a complementary motion cue for enhancement.Source: 3) Audio and radar fusion methods · p. 8

1) PIT-Based Separation Quality:

• RadarVox slightly outperforms DPRNN in multi-speaker PIT separation, with the three-speaker case remaining harder due to stronger interference and overlap.The reported trend is consistent with the two-speaker case; the authors describe radar motion cues as complementary information for separating overlapped speech.Source: 1) PIT-Based Separation Quality · p. 9, Table VI · p. 9

2) Speaker Identity Assignment:

• RadarVox achieves approximately 88% and 83% matching accuracy for two- and three-speaker mixtures, respectively, while substantially reducing the PIT-to-ordered gap.Ordered evaluation requires each separated stream to align with its corresponding radar-observed source.Source: 2) Speaker Identity Assignment · p. 9, Table VI · p. 9

D. Ablation Analysis

• A window size of 4, hop size of 1, and FFT size of 1024 provide a favorable trade-off between global temporal structure and transient speech cues.The preprocessing analysis uses window/hop heatmaps and an FFT-size evaluation.Source: D. Ablation Analysis · p. 9, Figure 9 · p. 9, Figure 10 · p. 9
• Radar integration consistently improves noise suppression, multi-speaker separation, intelligibility, and perceptual quality under PIT evaluation.Table VII reports the Stage-I radar integration ablation.Source: D. Ablation Analysis · p. 9, Table VII · p. 9
• In the two-speaker case, Stage-I radar fusion improves ordered SI-SDR from -6.44 dB to -0.03 dB, but its ordering bias is not explicit identity assignment.For three speakers, weak Stage-I ordering cues are insufficient to establish stable correspondence across the full three-way permutation.Source: Table VIII · p. 9, D. Ablation Analysis · pp. 9–10
• The matcher achieves over 80% accuracy in both two- and three-speaker cases using frozen Stage-I outputs and a PIT-best pseudo ground-truth ordering.Table IX evaluates Stage-II matcher learning under ordered evaluation.Source: D. Ablation Analysis · p. 10, Table IX · p. 10

VI. CONCLUSION

• The conclusion presents RadarVox as a publicly available radar-audio dataset and a two-stage framework integrating radar-gated fusion with explicit speaker assignment.The paper concludes that radar-derived motion cues can serve as complementary physical information for identity-aware speech perception in embodied voice-interaction environments.Source: §VI · p. 10