Source-linked AI summary
RDSE: Real-Time Directional Speech Enhancement Framework for Dual-Mic Commodity Smartphones
Chinmay Deo, Danish Mahajan, Soudeep Ghoshal, Himanshu Buckchash
TL;DR
Audio-visual zooming on mobile devices is constrained by dual-microphone hardware and the geometry and generalization assumptions of existing methods. RDSE combines one-time visual intent specification with hybrid spatial-neural processing; evaluations report intelligibility gains and competitive perceptual quality on SpARZ100 and real smartphone recordings, with real-time feasibility.
Problem
Mobile audio-visual zooming is constrained by limited spatial resolution in dual-microphone arrays and by existing methods' geometry, training-data, and generalization assumptions.
Source: §1 · p. 1Method
RDSE combines one-time camera-based direction specification with a cascade of dereverberation, MVDR beamforming, Conv-TasNet extraction, and MetricGAN post-filtering for dual-microphone smartphones.
Source: §1 · p. 2Results
Evaluations on SpARZ100 and real smartphone recordings report consistent intelligibility gains, competitive perceptual quality relative to strong neural baselines, and real-time feasibility.
Source: §1 · p. 2, §9 · p. 15Takeaways & Limitations
RDSE is presented as a practical audio-zooming framework for commodity dual-microphone devices that does not require precise array calibration or continuous visual conditioning.
Source: §9 · p. 15The signal model assumes far-field propagation, described as reasonable for typical conversational distances relative to microphone spacing.
Source: §3 · p. 3
Abstract
from arXiv · showhide
Audio-visual zooming on mobile devices is fundamentally constrained by limited microphone count, compute resource, unknown array geometry, and weak assumptions on real-world acoustic conditions. We present RDSE: Real-Time Directional Speech Enhancement Framework, a hybrid pipeline that combines traditional signal processing and modern machine learning methods for dual-microphone low-end smartphones. Unlike tightly coupled AV systems that continuously use visual feeds or embeddings, RDSE decouples visual interaction from feature inference, using camera input solely to parameterize the steering vector for spatial filtering. The proposed system uses four modules: dereverberation network, a beamformer with a temporal convolutional network, and a MetricGAN based denoiser. Extensive evaluation on benchmark data and real-world recordings demonstrates effective zooming without precise geometry calibration. Comparative experiments show competitive performance to existing methods. Proposed system is rigorously tested for statistical validity, ablation study, latency profiling, and practicality through real-time live-streaming.
1 Introduction
Audio-visual zooming aims to isolate speech from a chosen direction, but dual-microphone smartphones and existing spatial or learned approaches face constraints from limited spatial resolution, array mismatch, and generalization. RDSE combines one-time visual intent specification with geometry-tolerant spatial-neural processing, and is evaluated on benchmark data and smartphone recordings.
1 Introduction
- Dual-microphone arrays have limited spatial resolution and are sensitive to covariance errors and geometry mismatch, while learned approaches may assume fixed arrays or matched training data and generalize poorly.These constraints motivate combining classical beamforming with modular neural refinement without precise calibration or continuous visual feature extraction.Source: §1 · p. 1
- RDSE cascades channel-wise dereverberation, user-steered MVDR beamforming, Conv-TasNet target-speaker extraction, and MetricGAN post-filtering.The dereverberation module stabilizes spatial covariance estimation under reverberation; the post-filter suppresses residual noise.Source: §1 · p. 2
- Evaluations use benchmark datasets and real-world smartphone recordings, and a live-streaming implementation tests practical viability with intensive neural processing offloaded to cloud-based or offline processing.The reported evaluation demonstrates zooming without precise geometry calibration.Source: §1 · p. 2
- RDSE specifies listening direction through one-time camera-guided alignment or tap selection, decoupling visual control from audio inference and avoiding visual embeddings or language dependence.The study evaluates overlay-guided mode; tap-to-enhance is retained as a conceptual extension.Source: §1 · p. 2
- SpARZ100 is a controlled evaluation dataset designed to isolate anechoic and reverberant conditions for mobile audio-zooming benchmarks.The contribution list describes the dataset as supporting systematic benchmarking.Source: §1 · p. 2
2 Related Works
Prior audio-visual enhancement combines visual cues with speech processing, while later work explores more stable visual representations and robustness. RDSE targets dual-microphone smartphones by using a one-time visual azimuth to initialize geometry-tolerant spatial processing.
2 Related Works
- Audio-visual speech enhancement uses facial or lip embeddings to condition spectrogram masking, with later work improving visual features and addressing visual corruption.Decoupled systems such as Multimodal SpeakerBeam use fixed enrollment embeddings instead of continuous visual streams for more stable runtime behavior.Source: §2 · pp. 2–3
- Frame-level visual conditioning is vulnerable to occlusion, pose variation, and domain shift in mobile scenarios [12] [14].The cited passage introduces decoupled architectures as a robustness direction.Source: §2 · p. 2
- Existing multi-channel AV systems combine spatial filtering with neural refinement but assume calibrated arrays with at least four microphones, while mobile-efficient models emphasize computational compression over geometry constraints.Camera-guided prototypes also rely on calibrated arrays and continuous visual inference; recent edge-oriented designs retain frame-level visual processing.Source: §2 · p. 3
- RDSE addresses low-to-mid range dual-microphone smartphones by initializing geometry-tolerant spatial processing with a single user-guided azimuth, after which audio processing operates independently of the visual stream.Source: §2 · p. 3
3 Signal Model
The signal model describes recordings from a dual-microphone smartphone as target speech, interfering sources, and additive noise filtered by their propagation channels. It assumes far-field propagation to relate direction of arrival to inter-microphone time difference.
3 Signal Model
- Each microphone records target speech, interfering sources, and additive noise, with target and interferer signals filtered by their respective channel impulse responses.The model is presented for a dual-microphone recording scenario representative of commodity smartphones.Source: §3 · p. 3
- The channel impulse response models line-of-sight and multipath propagation as paths with attenuation and microphone-specific delay.The effective channel length is denoted L_h.Source: §3 · p. 3
- Under a far-field assumption, the inter-microphone time difference of arrival depends on microphone displacement and source azimuth, supporting the frequency-domain steering vector for the two-microphone array.The passage describes this assumption as reasonable for typical conversational distances relative to microphone spacing.Source: §3 · p. 3
4 Methodology
RDSE turns dual-microphone smartphone audio into a single-channel output for a user-specified direction through cascaded spatial filtering and neural enhancement. Its dereverberation, speaker-extraction, and denoising stages are paired with a streaming beamforming core.
4.1 RDSE Architecture
- The pipeline transforms each microphone channel, applies U-Net dereverberation, performs MVDR beamforming, then refines the output with Conv-TasNet and MetricGAN.The output is a single-channel signal corresponding to the user-specified listening direction.Source: §4.1 · p. 3, Figure 2 · p. 4
4.1.1 Dereverberation
- Dereverberation runs independently on each channel before beamforming to suppress late reflections and improve spatial covariance estimation while preserving inter-channel phase differences.The deployable module is toggleable for heavy reverberation (RT60 > 0.7 s); it predicts a dereverberated log-magnitude spectrum and reuses reverberant-input phase for reconstruction.Source: §4.1.1 · p. 4
4.1.2 MVDR Beamforming
- MVDR minimizes beamformer output power while preserving a distortionless response in the target direction; diagonal loading mitigates ill-conditioning in reverberant and noisy conditions.The beamformed output is converted back to the time domain by inverse STFT.Source: §4.1.2 · pp. 4–5
4.1.3 Speaker Extraction
- A pretrained Conv-TasNet in a one-rest configuration extracts the dominant speaker from the beamformer output to address residual interference and artifacts.The stated sources of residual interference and artifacts include geometry mismatch and nonstationary noise.Source: §4.1.3 · p. 5
4.1.4 Denoising
- A pretrained MetricGAN network estimates a magnitude mask while preserving phase to improve perceptual quality and suppress non-speech components.Source: §4.1.4 · p. 5
4.2 Real-Time Deployment and Validation
- The streaming implementation captures dual-channel audio at 16 kHz and processes it in 40 ms blocks with 75% overlap; neural modules are deferred to offline or cloud post-processing.The streaming core uses frequency-domain MVDR with per-bin covariance updated by exponential averaging and a fixed, visually specified steering direction.Source: §4.2 · pp. 5–6
5 Experimental Setup
The experimental setup combines a controlled simulated dual-microphone dataset with domestic smartphone recordings. The simulated dataset varies reverberation while holding other stated acoustic conditions constant.
5.1 Dataset Description
- SpARZ100 contains 100 simulated dual-microphone evaluation samples: 50 anechoic (RT60 = 0.0 s) and 50 reverberant (RT60 = 0.5 s), with SNR fixed at 5 dB.The two categories differ exclusively in acoustic condition; target utterances come from male LibriSpeech speakers and interfering utterances from female MUSAN speakers.Source: §5.1 · p. 6
5.2 Dataset Characteristics
- Across the dataset, reverberant samples have higher mixture energy, lower GCC-PHAT localization confidence, lower spectral centroid, and more dispersed ILD than anechoic samples.Anechoic samples show higher RMS-energy variance and more spread in GCC-PHAT sharpness; the figures plot these distributions across the two conditions.Source: §5.2 · pp. 6–7, Figure 5 · p. 7, Figure 6 · p. 7
5.3 Real-Time Data Capture Environment
- Real-world validation uses a Samsung F24 smartphone in a domestic indoor setting with male and female speakers at azimuths of 90° and 40°.The device’s microphone spacing is unknown and is approximated as 12 cm for beamforming and spatial-processing analysis.Source: §5.3 · p. 7
6 Evaluation Metrics
The evaluation reports intelligibility, perceptual speech quality, and speech quality using three objective metrics. Each metric compares processed audio with a clean or reference signal, with higher scores indicating better performance.
STOI
- STOI measures intelligibility by correlating short-time temporal envelopes from one-third-octave bands against the clean reference, with scores from 0 to 1.Source: §6 · p. 8
ViSQOL
- ViSQOL assesses perceptual speech quality by comparing spectro-temporal patches between reference and degraded signals, with scores from 0 to 5.Source: §6 · p. 8
PESQ
- PESQ estimates speech quality by modeling human auditory perception; its typical score range is −1 to 4.5.The evaluation integrates PESQ through a compiled reference implementation and custom MATLAB wrapper.Source: §6 · p. 8
7 Results and Discussion
RDSE shows strong anechoic performance and competitive results against prior methods, with quality-intelligibility trade-offs across pipeline configurations. Real-world recordings and latency measurements indicate near-real-time operation, while reverberation remains a challenge.
7.1 Perceptual Quality and Intelligibility
- Anechoically, RDSE scores STOI 0.7754, ViSQOL 2.0053, and PESQ 2.5755; performance degrades across all three metrics under reverberation.The reverberant condition is RT60 = 0.5 s; the passage attributes the degradation to spatial covariance errors from room reflections.Source: §7.1 · p. 8
7.2 Neural Backbone Selection
- Conv-TasNet is selected for RDSE, balancing enhancement performance and efficiency with approximately 3.8× lower inference time than SepFormer.SepFormer has the highest PESQ and ViSQOL scores, while Conv-TasNet's STOI performance is mixed across conditions.Source: §7.2 · pp. 8, 10, Table 2 · p. 10
7.3 Comparison with Prior Work
- RDSE achieves competitive intelligibility and perceptual quality in qualitative comparisons, with a dual-mic design trade-off against methods using different datasets or more microphones.The prior-work comparisons are not direct benchmarks; Li et al. report higher STOI under different conditions, and Nair et al. use a six-microphone array.Source: §7.3 · p. 10, Table 3 · p. 11
7.4 Ablation Studies
- Anechoically, adding MetricGAN raises PESQ to 2.58 from 1.91 while STOI falls marginally; reverberantly, Dereverb-off scores PESQ 1.82 versus the baseline's 1.87.Dereverb-on yields modest ViSQOL gains and competitive STOI, but mixed PESQ effects across conditions at higher computational cost; under reverberation, Denoising-off has the highest STOI (0.5592) but PESQ falls to 1.34.Source: §7.4 · pp. 10–11, Table 4 · p. 11
7.5 Real-World Validation
- In a Samsung F24 indoor recording with unknown geometry, the processed output attenuates the off-axis speaker while preserving target speech structure.The recording includes a male target at 90° and a female interferer at 40°; the pipeline maintains stable separation despite uncontrolled acoustics.Source: §7.5 · pp. 11–12, Figure 8 · p. 12
7.6 Real-time Latency
- On a 20.09 s live recording, the streaming core runs at RTF 0.60, but cloud neural refinement raises combined runtime to 20.87 s (RTF ≈1.04).Median and 95th-percentile frame times are 4.27 ms and 7.27 ms, both below the 8 ms hop duration.Source: §7.6 · p. 12, Table 5 · p. 12, §7.5 · p. 12
8 Statistical Analysis
Statistical tests find differences among the six methods across every metric and condition, while pairwise results show RDSE variants generally outperforming the baselines on STOI. Reverberation significantly reduces STOI for all RDSE variants.
8 Statistical Analysis
- The analysis combines descriptive statistics, a global test, post-hoc pairwise comparisons, and paired within-method tests, using α = 0.05.Bootstrap intervals are based on 1000 resamples and are used without distributional assumptions.Source: §8 · p. 12, §8.1 · p. 13, §8.2 · p. 13
8.2 Across-Method Comparison Using Friedman Test
- All six Friedman tests are significant (p ≪0.001), rejecting method equivalence across every metric and condition; PESQ has the highest Kendall’s W.The tests use the same 50 samples across methods; Table 6 reports test statistics, p-values, and effect sizes.Source: Table 6 · p. 13, §8.2 · p. 13
8.1 Descriptive Statistics and Confidence Intervals
- RDSE (MVDR + Conv-TasNet) leads STOI in both conditions (anechoic M = 0.789; reverberant M = 0.568), while RDSE (Dereverberation-off) leads PESQ in both.Bootstrap 95% CIs for STOI are [0.759, 0.812] anechoic and [0.539, 0.593] reverberant; dereverberation-on has PESQ M ≈0.88 in both conditions.Source: §8.1 · p. 13, Figure 9 · p. 14
8.3 Pairwise Post-Hoc Comparisons Using Nemenyi Test
- Under anechoic conditions, all RDSE variants significantly outperform both baselines on STOI (all p < 0.015); under reverberation, RDSE-baseline differences persist but intra-RDSE comparisons are non-significant.Anechoically, MVDR + Conv-TasNet is significantly superior to Denoising-off (p = 0.0099); other intra-RDSE differences are not significant.Source: Table 7 · p. 13, §8.3 · p. 15
8.4 Impact of Reverberation: Wilcoxon Signed-Rank Test
- All four RDSE variants show large, significant STOI degradation under reverberation (all p < 10^-9, r > 0.87); Dereverberation-off has the largest PESQ degradation.Dereverberation-on shows no significant PESQ change (p = 0.632), which the passage attributes to a floor effect.Source: §8.4 · p. 15
9 Conclusion
RDSE combines one-time visual intent specification with hybrid spatial-neural processing for dual-microphone smartphones without precise array calibration. Evaluations on SpARZ100 and real recordings report intelligibility gains, competitive perceptual quality, and real-time feasibility.
9 Conclusion
- RDSE combines one-time visual intent specification with hybrid spatial-neural processing for constrained dual-microphone smartphones.The conclusion reports operation without precise array calibration, continuous visual conditioning, or speaker/language dependence, and describes evaluation on SpARZ100 and real smartphone recordings.Source: §9 · p. 15