Multiclass Speech Classification Under Noise Disparity

1. INTRODUCTION

• Speech enhancement is a natural preprocessing candidate, but artifacts that vary with original noise, SNR, or enhancement model can replace one spurious cue with another.Suppressing class-discriminative noise therefore may not remove the shortcut.Source: §1 · p. 1
• The multiclass work extends prior binary noise augmentation, where opposite-class noise was used to expose both classes to comparable noise characteristics.In multiclass classification, a sample may be paired with noise from several other classes.Source: §1 · p. 1
• Noise disparity can induce class-discriminative cues that lead models to rely on noise rather than task-relevant speech information.The paper studies this problem in multiclass speech classification.Source: §1 · p. 1
• Augmenting each sample with noise from every other class suppresses the noise-disparity shortcut across SNR conditions, while random and batch-pairing strategies are ineffective.The paper compares three cross-class noise augmentation schemes.Source: §1 · p. 1
• Speech enhancement can introduce noise- and enhancement-dependent artifacts that impair classification under noise disparity.The artifacts may themselves become predictive of class labels.Source: §1 · p. 1

2. PRELIMINARIES

• A noisy sample y_c^i(k) is the ith sample from class c at time index k, modeled using clean speech and noise components.Only the noisy mixture is observed; the clean speech and noise components are unavailable.Source: §2 · p. 2
• Noise disparity means that the noise distribution depends on class, so different classes have different noise distributions that can provide spurious classification cues.The text connects this setting to recordings collected under different acoustic conditions.Source: §2 · p. 2
• The preliminaries define average power for a length-K signal and use signal powers and SNRs as notation throughout the paper.The time index is omitted in later notation for readability.Source: §2 · p. 2

• Cross-augmentation combines each sample with noise from the other class, matching noise characteristics so classifiers must rely on genuine speech content.The approach aims to remove the association between noise and class labels.Source: §3 · p. 2
• Noise is estimated from non-speech segments identified by a voice activity detector, and scalar coefficients scale the estimates added to each sample.The coefficients are selected to match SNR with respect to individual noises and total noise.Source: §3 · p. 2
• The two augmented samples use uniquely defined scaling coefficients that ensure matched noise characteristics.The matching conditions require equal SNR with respect to each individual noise and the total noise.Source: §3 · p. 2

• Augmenting each sample with noise from every other class exposes all samples to the same noise sources and matches noise characteristics, but lowers effective SNR as classes increase.This extension satisfies conditions for matching SNR across individual and combined noise sources.Source: §4 · p. 2
• Random single-noise augmentation adds noise from one randomly selected other class per epoch without SNR degradation as class count grows, but does not guarantee suppression of noise disparity.It does not satisfy all possible matching conditions in the multiclass setting.Source: §4 · pp. 2–3
• Batch pairing cross-augments samples from different classes using symmetric two-class matching, assigning unpaired samples noise from a randomly chosen other-class sample.The variant tests whether exact mutual noise matching for some class pairs improves on weaker additive exposure.Source: §4 · p. 3
• Speech enhancement is evaluated as preprocessing to suppress noise before classification rather than balance noise characteristics across classes.The study applies discriminative CR and generative SB enhancement models to noisy samples before training.Source: §4 · pp. 2–3

5.1. Databases

• The controlled dataset adds selected environmental noises to clean speech, keeping the underlying speech unchanged while introducing noise disparity.IEMOCAP supplies four emotion classes; four DEMAND noise types are each assigned to one class, at 20 dB, 10 dB, or randomly selected SNRs from 10 to 20 dB.Source: §5.1 · p. 3

5.2. Speech enhancement models

• The enhancement comparison uses discriminative CR and generative SB models, with best checkpoints trained on the WSJ0-CHiME3 benchmark.Both enhancement models follow a U-Net architecture.Source: §5.2 · p. 3

5.3. Training Protocol

• Emotion classification uses last-layer wav2vec 2.0 base representations, projected through two linear layers, and is evaluated by unweighted accuracy in leave-one-session-out cross-validation.Each fold holds out one session for testing; the other four are split into training and validation sets at 0.8 and 0.2.Source: §5.3 · p. 3
• Training uses Adam with a learning rate of 1 × 10−3 and selects the best model by validation loss, stopping by 100 epochs or a learning-rate threshold.The threshold is 10−4 of the initial learning rate.Source: §5.3 · p. 3

6. EXPERIMENTAL RESULTS

• Augmentation was assessed with oracle ground-truth noise and a practical estimate obtained from non-speech segments identified by voice activity detection.The oracle setting isolates augmentation from noise-estimation errors.Source: §6 · p. 3
• Augmenting each sample with noise from every other class closed the noisy–clean performance gap with oracle noise and at 20 dB in the practical setup.At 10 dB and random SNR, practical augmentation reduced but did not fully close the gap.Source: Table 1 · p. 4, §6 · p. 4
• Speech enhancement produced low noisy- and clean-test performance but substantially higher enhanced-test performance, including above the 59.3% clean-data reference.The authors attribute the enhanced-test advantage to noise-dependent artifact cues preserved or introduced by enhancement.Source: Table 2 · p. 4, §6 · p. 4
• The evaluation compares noisy and clean test sets, using their performance gap as the main measure of spurious-cue exploitation; clean-data training reached 59.3% UA.Enhanced test inputs were also evaluated for SE-based models.Source: §6 · pp. 3–4
• Randomly selected and batch-paired cross-augmentation left a substantial noisy–clean gap because epoch-varying added noise left original class-dependent noise cues available.The authors argue this encourages invariance to augmented noise while the original class noise remains fixed.Source: §6 · p. 4

7. CONCLUSION

• The study introduced controlled noise disparity into IEMOCAP to evaluate multiclass speech classification.Source: §7 · p. 4
• Among three cross-augmentation strategies, augmenting each sample with noise from every other class provided the strongest robustness.Source: §7 · p. 4
• Speech enhancement preprocessing did not resolve noise disparity and instead introduced noise- and enhancement-specific artifacts.Source: §7 · p. 4