Source-linked AI summary
SEANet: A Multi-modal Speech Enhancement Network
Marco Tagliasacchi, Yunpeng Li, Karolis Misiunas, Dominik Roblek
TL;DR
Conventional speech enhancement methods rely on noise assumptions that can limit performance in challenging conditions. SEANet conditions waveform denoising on earbud accelerometer data alongside microphone audio. The paper reports a 9.6 dB SI-SDRi improvement when the interferer is mixed at unit gain.
Problem
Traditional speech enhancement methods assume conditions such as known SNR or stationary noise, limiting their ability to handle challenging in-the-wild noise.
Source: §1 · p. 1Method
SEANet uses microphone and earbud accelerometer waveforms as input to a fully convolutional denoiser trained with adversarial and reconstruction losses.
Source: §1 · p. 1Results
9.6 dB SI-SDRi improvement was observed when the interferer was mixed with unit gain.
Source: §1 · p. 1Takeaways & Limitations
Earbud accelerometer data provides a strong conditioning signal, especially for denoising speech with overlapping speakers.
Source: §4 · p. 4The method assumes microphone audio, accelerometer data, and clean targets are time-aligned and available at the same sampling rate; lower-rate accelerometer data is interpolated.
Source: §2 · p. 2
Abstract
from arXiv · showhide
We explore the possibility of leveraging accelerometer data to perform speech enhancement in very noisy conditions. Although it is possible to only partially reconstruct user's speech from the accelerometer, the latter provides a strong conditioning signal that is not influenced from noise sources in the environment. Based on this observation, we feed a multi-modal input to SEANet (Sound EnhAncement Network), a wave-to-wave fully convolutional model, which adopts a combination of feature losses and adversarial losses to reconstruct an enhanced version of user's speech. We trained our model with data collected by sensors mounted on an earbud and synthetically corrupted by adding different kinds of noise sources to the audio signal. Our experimental results demonstrate that it is possible to achieve very high quality results, even in the case of interfering speech at the same level of loudness. A sample of the output produced by our model is available at https://google-research.github.io/seanet/multimodal/speech.
1. Introduction
The introduction motivates speech enhancement for noisy conditions and proposes conditioning waveform denoising on earbud accelerometer data. SEANet combines microphone and accelerometer waveforms, with an SI-SDRi improvement reported for speech interference at unit gain.
1. Introduction
- Traditional enhancement methods rely on assumptions such as known SNR or stationary noise, limiting their ability to handle challenging in-the-wild conditions.Source: §1 · p. 1
- Prior multimodal enhancement work used mouth or facial video, while other work combined audio and visual modalities or used multimodal cues for voice activity detection.Source: §1 · p. 1
- The proposed approach conditions speech denoising on an earbud bone-conductance accelerometer that captures local voice vibrations and is relatively insensitive to external sources.Earlier work used inertial sensors for speech recognition or reconstruction, and combined microphone and bone-sensor signals for denoising.Source: §1 · p. 1
- SEANet maps microphone and accelerometer waveforms directly to denoised speech using a fully convolutional model trained with adversarial and reconstruction losses.The adversarial losses encourage outputs that a discriminator cannot distinguish from clean speech; reconstruction losses preserve speech content while suppressing noise.Source: §1 · p. 1
- 9.6 dB SI-SDRi improvement was observed when the interferer was mixed with unit gain.The experiments consider overlapping speakers and background noise sampled from Freesound.Source: §1 · p. 1
2. Method
SEANet maps synchronized microphone and accelerometer waveforms to denoised speech with a fully convolutional generator. Its training combines adversarial and reconstruction losses, with a multi-resolution discriminator.
2. Method
- The generator takes microphone audio and accelerometer readings as inputs and produces a single-channel denoised speech waveform, assuming time-aligned signals at the same sampling rate.Accelerometer signals are interpolated before model input when their sampling rate is lower.Source: §2 · p. 2
- The UNet generator uses a symmetric encoder-decoder with skip connections, four blocks per side, and downsampling factors of (2, 2, 8, 8).Decoder blocks use transposed 1D convolution and residual units with dilation rates 1, 3, and 9; the encoder mirrors this structure.Source: §2 · p. 2
- Three structurally identical discriminators assess audio at original, 2x down-sampled, and 4x down-sampled resolutions, producing segment-level plausibility logits.Each logit judges the plausibility of an input segment corresponding to its receptive field.Source: §2 · p. 2
- SEANet trains its generator and discriminators with adversarial and reconstruction losses, using discriminator-layer feature differences as the reconstruction loss.The feature loss is less sensitive to small misalignment than per-sample waveform losses.Source: §2 · pp. 2–3
- Training uses reconstruction-loss weight λ = 100, three discriminator scales, Adam, batch size 16, and 200k iterations, or 2M for Librispeech.The learning rate is 0.0001; the final checkpoint is evaluated without parameter tuning or early stopping.Source: §2 · p. 3
3. Experiments
SEANet was evaluated on in-house and synthetically generated multimodal data under mixed-speech and background-noise conditions. Results show strong SI-SDRi, while accelerometer sampling-rate reductions affect speech mixtures more sharply than background-noise mixtures.
3. Experiments
- The in-house set contains about 1.25 hours from 25 speakers; microphone audio is 16kHz and one 4kHz accelerometer axis is interpolated to 16kHz.Signals were high-pass filtered at 20Hz and normalized and clipped to handle isolated accelerometer spikes.Source: §3 · p. 3
- Evaluation mixes clean speech with another speaker or Freesound noise, varies accelerometer bandwidth, and measures enhancement using SI-SDR.The synthetic Librispeech set follows the canonical train-clean-100 and test-clean split.Source: §3 · p. 3
- 8.9dB SI-SDRi for mixed noise and 9.6dB for mixed speech; audio-only reaches 7.9dB on mixed noise and cannot separate mixed speech.The in-house results were averaged across five replicas and five splits.Source: Table 1 · p. 4, §3 · p. 4
- 12.4dB SI-SDRi on both mixed noise and mixed speech was achieved on Librispeech using synthetically generated accelerometer signals.The signals were synthesized from Librispeech audio using a model trained on in-house audio–accelerometer pairs.Source: §3 · pp. 3–4, Table 2 · p. 4
- Below 400Hz, SI-SDRi drops rapidly for overlapping speakers, with separation failing below 200Hz; background-noise performance declines only slightly.Average SI-SDRi drops from 8.9dB to 8.0dB across splits and from 12.4dB to 9.8dB on Librispeech.Source: Figure 4 · p. 4, §3 · p. 4
4. Conclusions
Earbud-mounted accelerometer data provides a strong conditioning signal for speech denoising, especially with overlapping speakers. The paper proposes exploring additional microphone, accelerometer-axis, and visual inputs.
4. Conclusions
- Earbud accelerometer data provides a strong conditioning signal for speech denoising.Source: §4 · p. 4
- The conditioning signal is especially useful when speakers overlap.Source: §4 · p. 4
- Future work will explore combining multiple microphones, accelerometer axes, and visual cues in SEANet.Source: §4 · p. 4