Joint Optimization of Masks and Deep Recurrent Neural Networks for Monaural Source Separation

Po-Sen Huang; Minje Kim; Mark Hasegawa-Johnson; Paris Smaragdis

Source-linked AI summary

Joint Optimization of Masks and Deep Recurrent Neural Networks for Monaural Source Separation

Po-Sen Huang, Minje Kim, Mark Hasegawa-Johnson, Paris Smaragdis

arXiv:1502.04149v4Feb 13, 2015cs.SDcs.AIcs.LGcs.MM

TL;DR

Monaural source separation is underdetermined from a single channel, motivating models that impose reconstruction constraints. The paper jointly optimizes time-frequency masks with deep recurrent networks and adds discriminative training, achieving gains across speech separation, singing voice separation, and speech denoising tasks.

Problem
Monaural source separation is challenging because, without constraints, a single-channel mixture has infinitely many possible solutions.
Method
The paper jointly optimizes time-frequency masking functions as a layer with deep recurrent neural networks and explores a discriminative training criterion.
Results
The proposed models improve performance across speech separation, singing voice separation, and speech denoising, including 2.30–2.48 dB GNSDR gain and 4.32–5.42 dB GSIR gain over RNMF in singing voice separation.
Takeaways & Limitations
Joint mask optimization with recurrent modeling and discriminative training provides a general framework for monaural source separation across multiple tasks and conditions.

Abstract

from arXiv · show

Monaural source separation is important for many real world applications. It is challenging because, with only a single channel of information available, without any constraints, an infinite number of solutions are possible. In this paper, we explore joint optimization of masking functions and deep recurrent neural networks for monaural source separation tasks, including monaural speech separation, monaural singing voice separation, and speech denoising. The joint optimization of the deep recurrent neural networks with an extra masking layer enforces a reconstruction constraint. Moreover, we explore a discriminative criterion for training neural networks to further enhance the separation performance. We evaluate the proposed system on the TSP, MIR-1K, and TIMIT datasets for speech separation, singing voice separation, and speech denoising tasks, respectively. Our approaches achieve 2.30--4.98 dB SDR gain compared to NMF models in the speech separation task, 2.30--2.48 dB GNSDR gain and 4.32--5.42 dB GSIR gain compared to existing models in the singing voice separation task, and outperform NMF and DNN baselines in the speech denoising task.

I. INTRODUCTION

Monaural source separation recovers multiple original signals from one mixed recording, but is underdetermined without prior knowledge. The paper motivates nonlinear, domain-agnostic deep models and proposes a jointly constrained framework for several separation tasks.

Monaural source separation is challenging because a single-channel mixture admits infinitely many solutions without prior knowledge.
Domain-specific methods use task assumptions, whereas domain-agnostic methods learn directly from data without prior assumptions.
NMF and PLSI provide linear transformations, while DNNs are desirable because mixture-to-source mappings can be nonlinear.
The proposed framework jointly models all mixture sources with a DRNN and learns time-frequency masks to enforce mixture-output constraints.
The framework is evaluated for speech separation, singing voice separation, and speech denoising using the TSP, MIR-1K, and TIMIT datasets.

III. PROPOSED METHODS

The proposed methods use deep recurrent architectures to model temporal information in monaural source separation. They compare recurrence at one hidden layer with recurrence at all levels and describe the associated computational costs.

Deep Recurrent Neural Networks: Deep recurrent neural networks model temporal information in audio signals for monaural source separation.
Deep Recurrent Neural Networks: DRNN-l places recurrent connections at the l-th layer of an L-layer network, while stacked RNNs use recurrent connections at all levels.
Deep Recurrent Neural Networks: The DRNN formulation uses layer-specific nonlinear transformations, feedforward weights, recurrent weights, and a linear output layer.
Deep Recurrent Neural Networks: A DNN is equivalent to a DRNN whose temporal weight matrix U_l is a zero matrix.
Deep Recurrent Neural Networks: Compared with a DNN, a DRNN adds Θ(m^2) storage and Θ(m^2) multiply-add operations for a recurrent connection during forward propagation.

B. Model Architecture

The model predicts multiple source spectra from mixture features, then applies jointly optimized soft time-frequency masks. This masking layer smooths estimates and enforces consistency between predictions and the original mixture.

The framework considers two additively mixed sources and uses mixture logmel features or magnitude spectra as network inputs.
The network simultaneously predicts magnitude spectra for both sources rather than training on only one source.
The soft mask assigns complementary weights to the two source predictions and applies them element-wise to the mixture magnitude spectrum.
The masking functions are added as a neural-network layer and jointly optimized with the deep learning model.
The added deterministic layer integrates reconstruction constraints while network weights are trained by back-propagation against source targets.

C. Training Objectives

The training objective begins with squared error reconstruction and adds discriminative training to reduce interference between competing sources. The criterion uses a swapped-source likelihood comparison and may trade reduced interference for increased artifacts.

Squared error minimizes the difference between predicted and actual source targets.
The MSE objective can produce conservative predictions that assign ambiguous spectral features to both sources, reducing testing SIR through interference leakage.
The discriminative term is a point estimate of KL divergence between the original-source likelihood model and a model with source affiliations swapped.
The discriminative criterion adds a regularization term that penalizes predictions resembling the other source.
The discriminative criterion targets lower interference and may increase artifacts, while experiments generally report higher SIR with similar or higher SDR and SAR.

IV. EXPERIMENTS

The experiments evaluate monaural separation across speech separation, singing voice separation, and speech denoising using standard BSS-EVAL measures. Results and example outputs compare network variants, discriminative training, and NMF settings.

The evaluation covers speech separation, singing voice separation, and speech denoising tasks.
SDR measures overall separation performance, while SIR measures residual interference and SAR measures introduced artifacts.
DRNN-k denotes a recurrent connection at the k-th hidden layer, whereas sRNN has recurrent connections at all hidden layers.
The NMF comparison in the TSP results uses spectral features.
Models are trained by back-propagation using the Eq. (11) objective and L-BFGS from random initialization.

A. Speech Separation Setting

The TSP speech-separation experiments use mixtures of selected female and male speakers, with train, development, and test splits. They compare neural models with speaker-specific NMF baselines across feature types and mixing cases.

The TSP corpus contains 1,444 utterances averaging 2.372 seconds from 24 speakers.
Four speakers—FA, FB, MC, and MD—form three mixing cases: FA–MC, FA–FB, and MC–MD.The same-gender FA–FB and MC–MD cases are expected to be more difficult.
The data are split into 80% training, 10% development, and 10% testing, with signals downsampled to 16 kHz.
NMF uses 10–100 speaker-specific basis vectors, with the optimal number selected on the development set.Twenty basis vectors achieved the best development performance across the three mixing cases.
Figures 8 compares male-versus-male TSP separation results across joint-training and discriminative-training conditions.
The experiments compare spectral and log-mel filterbank input features.The spectral representation uses a 1024-point STFT with 50% overlap; the log-mel representation is 40-dimensional with derivative features.

B. Speech Separation Results

Speech separation experiments compare NMF with neural models across speaker-gender conditions, architectures, features, masking optimization, and discriminative training. Joint masking optimization and suitable recurrent or log-mel configurations generally improve separation, with especially large gains over NMF in same-sex cases.

Masking and features: Jointly optimizing the masking layer with the DNN significantly outperforms applying the masking layer separately.
Masking and features: In joint optimization cases, log-mel features achieve higher SDR, SIR, and SAR than spectral features.
Network architectures: Recurrent models, including DRNN-1, DRNN-2, and sRNN, outperform DNN in many speech-separation cases.
Discriminative training: Discriminative training improves SIR in most cases but can slightly lower SAR because of artifacts; γ is empirically set between 0.01 and 0.1.
Comparison with NMF: The proposed models greatly outperform NMF across female-male, female-female, and male-male separation conditions.
Comparison with NMF: Around 5 dB SDR gain over NMF is achieved in the FA versus FB case while maintaining higher SIR and SAR.

C. Singing Voice Separation Setting

The singing voice separation setting treats singing voice and background music as two sources to separate from mixtures. Experiments use MIR-1K clips, fixed train/development and test partitions, equal-energy mixtures, and magnitude-spectrum inputs.

Task and dataset: The task separates singing voice from background music in monaural music recordings.
Task and dataset: MIR-1K contains 1,000 Chinese karaoke clips sampled at 16 kHz and lasting 4 to 13 seconds.
Experimental split: The training and development set uses 175 clips from one male and one female singer, while 825 clips from 17 singers form the test set.
Experimental split: Singing voice and music are mixed with equal energy, corresponding to 0 dB SNR.
Evaluation: Performance is reported using Global NSDR, Global SIR, and Global SAR, weighted across test clips by clip length.
Evaluation: NSDR measures SDR improvement between the separated singing voice and the mixture relative to the clean singing voice.
Input features: Magnitude spectra are extracted with a 1024-point STFT and 50% overlap; log-mel and log-power features perform worse empirically.

D. Singing Voice Separation Results

Singing voice separation results examine output design, recurrent architectures, discriminative training, and comparisons with prior approaches. Modeling both sources and jointly optimizing masking improves results, while discriminative training primarily strengthens interference suppression.

Output modeling: Modeling two sources simultaneously provides higher GNSDR, GSIR, and GSAR than using a single source as the target.
Output modeling: Jointly optimizing the masking layer and DRNN further improves separation results over the corresponding non-joint configuration.
Discriminative training: Discriminative training further improves GSIR while maintaining similar GNSDR and GSAR across the evaluated architectures.
Comparison with prior work: The proposed models achieve 2.30–2.48 dB GNSDR gain and 4.32–5.42 dB GSIR gain over RNMF, with similar GSAR.

E. Speech Denoising Setting

The speech denoising experiments separate clean speech from noise using TIMIT utterances mixed with several noises at specified SNRs. Results compare NMF, DNN, and jointly optimized masking models under matched and multiple-SNR conditions.

Task and features: The task separates clean speech from noise in noisy speech recordings.
Dataset and mixtures: Experiments use 100 TIMIT utterances from 10 speakers and five noises: Airport, Train, Subway, Babble, and Drill.
Dataset and mixtures: Noisy mixtures are generated by overlaying speech with randomly selected subsets of noises at specified SNRs.
Evaluation: Figure 10 compares denoising architectures with and without discriminative training on inputs trained and tested at 0 dB SNR.
Evaluation: Figure 11 evaluates NMF, non-joint DNN, and jointly optimized masking DNN models when testing across multiple SNR inputs using a model trained at 0 dB SNR.

F. Speech Denoising Results

Speech denoising experiments compare NMF, DNN, and jointly optimized masking models across matched and mismatched conditions. Performance degrades most when both speakers and noise are unseen, while unseen speakers alone cause only mild degradation.

Architecture and training effects: Recurrent architectures achieve similar performance to the DNN model under 0 dB SNR inputs.The comparison includes DRNN-1, DRNN-2, and sRNN.
Architecture and training effects: Including the discriminative training objective improves SDR and SIR but slightly degrades SAR while leaving STOI similar.
Mismatch conditions: Speech denoising results compare NMF, DNN without joint masking optimization, and DNN with joint masking optimization on 0 dB SNR inputs.Figure 12 reports average values with minimum and maximum observed ranges for known and unseen speakers and noise.
Overall comparison: MASK achieves better performance than other targets in speech denoising tasks.
Mismatch conditions: Unseen noise causes larger degradation, and jointly unseen speakers and noise produce the worst performance among the tested conditions.The unseen noise is Drill noise, which differs substantially from the training noises in spectral and temporal structure.
Mismatch conditions: Unseen-speaker conditions produce only mild performance degradation across all compared models relative to known-speaker conditions.

G. Discussion

The discussion attributes improvements across three source-separation tasks to jointly optimized time-frequency masks, recurrent networks, and discriminative training. The models outperform baselines in matched and mismatched settings, while recurrent architectures add task-dependent gains at extra memory and computational cost.

Overall findings: Jointly optimizing time-frequency masks with deep recurrent neural networks and discriminative training improves results over baseline models across speech separation, singing voice separation, and speech denoising.The improvement is reported under various settings, including matched and mismatched conditions.
Recurrent architectures: Deep recurrent neural networks add 0.5 dB SDR gain in speech separation and 0.34 dB GNSDR gain in singing voice separation.These gains come with extra memory and computational costs compared with feed-forward neural networks.
Overall findings: 2.30–4.98 dB SDR gain is achieved over the NMF baseline in the TSP speech separation task.The proposed models also maintain higher SIRs and SARs.
Overall findings: 2.30–2.48 dB GNSDR gain and 4.32–5.42 dB GSIR gain are achieved over previously proposed methods in MIR-1K singing voice separation.GSARs remain similar.
Speech denoising: The proposed method outperforms NMF and DNN baselines in various mismatch conditions for TIMIT speech denoising.
Future work: Future work includes exploring LSTMs for longer temporal information and applying the models to applications such as robust ASR.The paper notes LSTM properties related to avoiding vanishing gradients.