PaperlayerView exact revision on arXiv

Source-linked AI summary

Multirate State Space Models for End-to-End Processing of Pulse Density Modulated Speech Signals

Ludovic Boulanger, Sean U. N. Wood

arXiv:2608.28472v1cs.SDeess.AS

TL;DR

The paper addresses the computational and sampling-rate limitations of direct PDM speech processing on low-power devices. It uses an SSM encoder to create a modulation- and sampling-rate-invariant representation that can be aggressively downsampled, achieving robust results at 512 kHz and performance near the PCM baseline at 2 MHz.

  • Problem

    PDM-to-PCM conversion adds filtering and decimation cost, while prior direct-PDM methods require long training sequences and generalize poorly across sampling rates.

  • Method

    An SSM encoder produces a modulation- and sampling-rate-invariant PDM representation that can be downsampled without anti-aliasing.

  • Results

    92.66% keyword-spotting accuracy and PESQ/STOI of 2.98/0.92 at 512 kHz, and 93.13% accuracy and PESQ/STOI of 3.08/0.93 at 2 MHz, all within 0.5% of the PCM baseline.

  • Takeaways & Limitations

    The architecture can be trained on short 16 kHz PCM sequences and deployed directly across tested PDM sampling rates without retraining.

Abstract

from arXiv · show

Deep neural networks (DNNs) based on state-space models (SSMs) are increasingly applied to speech processing, but typically operate on pulse-code-modulated (PCM) audio. This constrains deployment on low-power, always-on edge devices, which commonly use single-bit pulse-density-modulated (PDM) micro-electromechanical (MEMS) microphones for their noise robustness, low cost, and variable sampling rates that enable low-power operation. In fact, converting PDM to PCM requires low-pass filtering and decimation, imposing costly overhead on resource-constrained hardware. While prior works have attempted to process PDM signals directly, they require long training times and generalize poorly across sampling rates. In this paper, we show that the SSM has two key properties that remediate these issues: its continuous-time parametrization allows it to produce a consistent representation of the input audio signal, regardless of the modulation strategy and sampling rate, and its long-term memory enables this representation to be aggressively downsampled without needing any anti-aliasing operations. We then propose a novel end-to-end PDM speech processing architecture that uses an SSM to encode the input audio signal into a modulation- and sampling-rate-invariant latent representation. We show that our proposed architecture achieves robust speech classification and enhancement gains at low-power sampling-rates (512 kHz) and similar performance to state-of-the-art algorithms operating on PCM data when tested on standard PDM sampling-rates of 2 MHz. Moreover, we show that the SSM's output can be downsampled by more than 65,000 times, thus significantly reducing the number of processing timesteps in downstream layers.

I. INTRODUCTION

SSM-based speech models are usually designed for PCM audio, limiting their use on low-power edge devices with variable-rate PDM microphones. The paper proposes an SSM encoder that supports direct PDM processing across sampling rates while reducing downstream computation.

  • Motivation: PDM microphones support low-power edge deployment but conventional speech processing first converts PDM to PCM through costly filtering and decimation.PDM devices use variable sampling rates, so algorithms must also handle different input rates efficiently.
  • Limitations of prior approaches: Existing CNN and end-to-end PDM approaches avoid explicit conversion but require long, memory-intensive training sequences and often specialize to one sampling rate.Consequently, they cannot exploit multiple PDM power modes without separate training for each mode.
  • Proposed approach: The proposed architecture uses an SSM encoder to extract modulation-invariant features shared by PCM and PDM representations.Its continuous-time modeling is intended to produce a common representation across the two encodings.
  • Proposed approach: The SSM representation can be downsampled without anti-aliasing, making downstream layers independent of input modulation and sampling rate.This design targets the computational overhead of long PDM sequences while remaining compatible with different PDM operating modes.
  • Evaluation: The architecture is evaluated on keyword spotting and speech enhancement, with comparisons involving PCM-trained and PDM-processing systems.The paper describes validation on both a keyword-spotting task and a more complex speech-enhancement task.

II. BACKGROUND

PDM microphones provide low-power, variable-rate speech acquisition, but their single-bit oversampled output typically requires costly conversion before processing. SSMs model input histories with continuous-time dynamics and basis-function coefficients, supporting fixed-window or decaying memory representations that can be adapted for PDM processing.

  • A. Pulse Density Modulation: PDM microphones use oversampled single-bit streams whose one-density encodes waveform amplitude, while their sampling rates can vary across power modes.PDM devices can operate in sleep, low-power, and standard/performance modes with different sampling rates.
  • A. Pulse Density Modulation: PCM can be recovered from PDM through low-pass filtering and decimation because both encodings share frequency content below the PCM Nyquist frequency.Sigma-Delta oversampling and noise shaping push much of the quantization noise above the PCM signal band.
  • A. Pulse Density Modulation: Higher oversampling ratios make the Sigma-Delta power spectral density more closely follow PCM below the PCM Nyquist rate; at OSR 64, second-order Sigma-Delta follows PCM to 8 kHz.The comparison concerns the cleanliness of the signal band as OSR and modulator order change.
  • B. State Space Models: SSMs map an input sequence to high-dimensional state coefficients representing a continuous-time window of its history, with state dimension H determining the number of basis functions.The memory duration is controlled by the continuous-time parameter θ, and the coefficients support reconstruction over the interval [t − θ, t].
  • B. State Space Models: LegT and FouT use fixed continuous-time memory windows, whereas LegS represents the entire history with exponentially decaying memory accuracy.FouT uses scaled truncated Fourier bases, LegT uses scaled Legendre polynomials, and the windowed initializations support near-perfect reconstruction within their windows.
  • B. State Space Models: The proposed encoder converts PCM and PDM inputs into coefficient sequences that can be downsampled to a fixed rate, making downstream processing independent of modulation and sampling rate.For integer KPDM = OSR, decimated PDM coefficients encode the same continuous-time history as corresponding PCM coefficients, without anti-aliasing.

III. METHODS

The proposed architecture uses an SSM encoder to produce a modulation- and sampling-rate-invariant representation, allowing downstream processing on a unified sequence.

  • The architecture uses an SSM encoder layer to produce a representation invariant to modulation format and sampling rate.Downstream layers therefore operate on a single unified representation.

A. SSM Encoder Layer

The SSM encoder aligns PCM and PDM inputs by exploiting continuous-time memory and downsampling redundant coefficient sequences without anti-aliasing. FouT and LegT predominantly encode frequencies below the PCM Nyquist frequency, yielding similar responses for both modulations.

  • FouT and LegT basis functions predominantly encode frequencies below the 8 kHz PCM Nyquist frequency.This supports similar SSM responses to PCM and PDM speech inputs.
  • The encoder produces nearly identical coefficient trajectories for 16 kHz PCM and 1.024 MHz PDM speech, aside from PDM quantization noise.The PCM and PDM discretization steps are 1/16,000 and 1/1,024,000, respectively.
  • PDM generates SSM coefficients at a proportionally higher temporal rate than PCM, creating a rate mismatch for PCM-trained downstream layers.The mismatch can impair generalization to PDM-derived coefficients.
  • Each state vector summarizes a fixed continuous-time memory window, so PDM windows contain OSR times more samples than PCM windows.Adjacent coefficient vectors consequently summarize highly overlapping windows and contain temporal redundancy.
  • For integer OSR, PDM coefficients can be downsampled with hop size OSR without anti-aliasing while preserving the corresponding PCM continuous-time history.The downsampled coefficients remain aligned with the non-decimated PCM coefficient spectrogram.
  • PCM coefficient sequences can also be downsampled below the Nyquist rate to reduce downstream operations without increasing overall method complexity.The paper evaluates this strategy for keyword spotting and speech enhancement.

B. Proposed PDM Speech Processing Architecture

The proposed architecture feeds SSM coefficients into task-specific downstream networks for keyword spotting or speech enhancement, with enhanced PCM reconstructed from predicted coefficients.

  • The SSM encoder maps input speech to a coefficient sequence used by both keyword spotting and speech enhancement modules.The architecture supports both tasks within the same encoder framework.
  • For keyword spotting, a classifier converts SSM coefficients into keyword class probabilities.
  • For speech enhancement, a DNN predicts clean-speech SSM coefficients that are projected onto PCM-rate basis functions.Overlap-add combines the resulting time-domain frames into the enhanced waveform.

IV. EXPERIMENTAL SETUP

The experimental setup trains downstream networks on PCM data and evaluates whether they generalize to PDM inputs at various sample rates using the proposed architecture.

  • The proposed representation is evaluated for extending PCM-specific keyword-spotting and speech-enhancement DNNs to PDM data.The architecture is intended to avoid substantially increasing training time or memory requirements.
  • The SSM encoder creates a modulation- and sampling-rate-invariant representation for downstream task networks.Figure 7 depicts the classifier and speech-enhancement branches.
  • The DNNs are trained on PCM data and evaluated for generalization to PDM at various sample rates.

A. PCM Training

Training uses 16 kHz PCM data augmented with high-pass filtered noise to mimic PDM quantization noise across OSRs. PCM waveforms are also converted to synthetic PDM streams for evaluation at 128 kHz–2 MHz.

  • A. PCM Training: 16 kHz PCM training data is augmented with high-pass filtered white noise to improve generalization across PDM OSRs.The filter models quantization noise that varies with OSR and modulator order.
  • A. PCM Training: G is sampled log-uniformly from [10^-3, 10^-1], while β is fixed at 0.93 to match PDM spectral characteristics.β controls low-frequency noise leakage, and its value was empirically selected using power spectral density matching.
  • A. PCM Training: Speech-enhancement training additionally addresses the higher effective resolution of clean 16 kHz PCM targets relative to 128 kHz PDM inputs.The supplied passage introduces an augmentation strategy for learning this effective bit-rate increase, but does not provide its full procedure.
  • A. PCM Training: PCM data is interpolated and converted into single-bit second-order Σ∆ PDM streams using OSRs from 8 to 128.These OSRs correspond to sampling rates from 128 kHz to 2 MHz.
  • A. PCM Training: Keyword-spotting evaluation uses Google Speech Commands, containing 105,829 clean utterances from 2,618 speakers across 35 keyword classes.The input audio is provided in 16 kHz PCM format.

2) Downstream DNN:

The architecture combines an SSM encoder with task-specific downstream networks for keyword spotting and speech enhancement. The downstream models produce class logits or SSM coefficients used to reconstruct clean speech.

  • 2) Downstream DNN:: Keyword spotting uses six LegS-initialized SSM blocks with hidden dimension 64 and 64-dimensional input and output features.Each block contains a unidirectional SSM memory cell connected within the downstream stack.
  • 2) Downstream DNN:: Keyword-spotting outputs are averaged over time and projected to 35 classification logits, with the predicted class chosen by argmax.The downstream network therefore maps encoded sequences to one of the 35 keyword classes.
  • 2) Downstream DNN:: The VoiceBank+DEMAND training data contains 11,572 noisy utterances from 28 speakers mixed with ten noise types at 0–15 dB SNR.The dataset is used as the speech-enhancement benchmark.
  • 2) Downstream DNN:: Speech enhancement integrates LiSenNet, retaining under 40k parameters while removing Griffin-Lim and skip connections to reduce latency and model size.The model predicts SSM coefficients rather than directly predicting clean spectrogram magnitudes and phases.
  • 2) Downstream DNN:: The speech-enhancement decoder outputs SSM coefficients that reconstruct the clean PCM waveform through projection onto their basis functions.This replaces the original LiSenNet output representation with coefficients tied to the SSM encoder.

3) Training Procedure:

Training uses MetricGAN-based adversarial learning for speech enhancement, while experiments vary SSM memory, decimation, and state dimension before testing PCM-trained models across PDM rates.

  • 3) Training Procedure:: MetricGAN adversarial training uses a discriminator to predict PESQ and a generator to predict clean speech from noisy waveforms.PESQ is estimated from log-compressed STFT magnitudes of estimated and target speech.
  • 3) Training Procedure:: The modified enhancement model predicts clean SSM coefficients, whose synthesized PCM waveform is evaluated using a 512-point STFT with hop size 256.This preserves compatibility with the original LiSenNet training setup.
  • 3) Training Procedure:: Experiments vary the memory window θ, decimation factor K_PDM, and state dimension H to assess their effects on keyword spotting and speech enhancement.These parameters control encoded history, coefficient spacing, and the number of basis functions, respectively.
  • 3) Training Procedure:: The first two experiments use 2 MHz PDM data while varying θ and H, then K_PDM and θ, with H fixed at 32 in the latter setting.The 2 MHz rate represents a standard or high-resolution PDM microphone mode.
  • 3) Training Procedure:: A third experiment trains one fixed configuration on 16 kHz PCM waveforms and tests it across PDM sampling rates from 128 kHz to 2 MHz.This directly evaluates cross-sampling-rate generalization without retraining separate models.

1) Effect of Varying the Memory Window Length θ and State Dimension H on Classification Accuracy:

Classification remains strong with compact SSM configurations and aggressive coefficient decimation, while enhancement depends more strongly on SSM family and window-state alignment. The architecture generalizes from PCM training to multiple PDM rates and can achieve low latency.

  • Classification accuracy: 93.72% classification accuracy is achieved by LegT with H = 128 and θ = 8 ms, versus 93.44% for the best FouT configuration.The LegT result uses one quarter as many bases as the best FouT configuration.
  • Classification accuracy: LegT with H = 32 has nearly the same classification accuracy as H = 128 while requiring substantially less computation because the SSM A matrix scales quadratically with H.This identifies H = 32 as a promising lightweight configuration for constrained devices.
  • Classification accuracy: With H = 32 and an 8 ms memory window, classification accuracy remains 88% up to K_PDM = θ̄_PDM and 83% up to K_PDM = 4 × θ̄_PDM.Across all tested decimation factors, accuracy drops by less than 10%.
  • Classification accuracy: A 65,536-fold reduction at 2 MHz produces downstream coefficient sequences at 31.25 Hz without anti-aliasing operations.The SSM encoder enables further coefficient decimation while substantially reducing downstream neural-network operations.
  • Classification accuracy: PCM-trained models maintain over 92% classification accuracy above 512 kHz PDM sampling rates, corresponding to less than a 4% drop across those OSRs.The tested configuration uses H = 32 and θ = 2 ms and never sees PDM during training.
  • Speech enhancement: The best enhancement configuration is FouT with θ = 32 ms and H = 512, achieving PESQ 3.24 and STOI 93%.A compact θ = 2 ms, H = 32 configuration achieves PESQ 3.08 and STOI 93%, close to PCM LiSenNet’s PESQ 3.07 and STOI 94%.
  • Speech enhancement: FouT consistently outperforms LegT for PDM enhancement, with performance degrading when θ̄_PDM differs from H; PCM results show more similar SSM performance.The reported difference is linked to how each SSM’s basis-vector spectra cover frequencies relative to the PCM Nyquist frequency.
  • Speech enhancement: Greater overlap between consecutive enhancement windows improves denoising, with FouT reaching PESQ 3.13 at 87.5% overlap and 3.08 at 75%.The 87.5% setting yields 4.5 ms total algorithmic latency, below the cited 6 ms hearing-aid delay requirement.

3) Effect of PDM Sampling Rate on Speech Enhancement Performance:

Across PDM sampling rates, the proposed architecture remains above noisy-speech enhancement scores, with STOI reaching the noisy baseline at 512 kHz.

  • 2.24: the architecture’s lowest PESQ score across tested PDM sampling rates, at 128 kHz, exceeds the noisy speech PESQ score of 2.0.
  • 0.92: the noisy speech STOI score, already matched by the model at 512 kHz.

VI. CONCLUSION

The proposed SSM-based architecture supports PCM-only training and direct deployment on PDM signals across sampling rates without finetuning. It preserves performance near the PCM baseline in low-power and standard operating ranges, while future work targets higher FouT frequency resolution and embedded real-time validation.

  • The architecture can be trained solely on PCM sequences and deployed directly to PDM data at various sampling rates without retraining.
  • 92.66% keyword-spotting accuracy and PESQ/STOI of 2.98/0.92 are maintained at the 512 kHz low-power operating range.
  • 93.13% accuracy and PESQ/STOI of 3.08/0.93 are maintained at 2 MHz, all within 0.5% of the PCM baseline.
  • Future work will investigate focusing FouT basis vectors below the PCM Nyquist frequency to increase frequency resolution and possibly improve speech enhancement.
  • Future work will implement the algorithm on an FPGA connected to a PDM microphone to validate adaptation to sudden sampling-rate changes.
Loading 2608.28472v1…