Source-linked AI summary
U-PAST: A Phase-Aware Audio Spectrogram Transformer-U-Net for Single-Channel Speech Enhancement
Cao Duong Ly, Jörn Anemüller
TL;DR
Single-channel speech enhancement requires effective spectral and temporal modeling despite lacking spatial information, while conventional convolution captures long-range dependencies indirectly. U-PAST combines complex-spectrogram tokenization and transformer self-attention with a complex-valued U-Net decoder to reconstruct enhanced magnitude and phase. Across mismatch conditions, it achieves the best evaluated SI-SDR under acoustic mismatch while retaining a small compute and parameter footprint, although its wall-clock latency is not better than convolutional baselines.
Problem
Single-channel speech enhancement must recover clean speech while modeling spectral structure and temporal context without spatial information.
Method
U-PAST tokenizes complex STFT magnitude and phase, applies a multi-layer transformer encoder, and reconstructs the enhanced spectrogram with a complex-valued U-Net decoder.
Results
U-PAST achieves the best evaluated SI-SDR under acoustic mismatch and remains competitive elsewhere with 1.17M–2.40M parameters and substantially lower compute than stronger baselines.
Takeaways & Limitations
U-PAST offers a favorable accuracy-to-compute trade-off at a small parameter footprint, with U-PAST-H the strongest family variant.
Takeaways & Limitations
U-PAST’s favorable parameter and compute footprint does not translate into lower wall-clock latency than purely convolutional baselines.
Abstract
from arXiv · showhide
Convolutional neural networks (CNNs), used widely and successfully in audio enhancement, capture long-range time-frequency dependencies only indirectly, through successive convolution and pooling. Here, we present U-PAST, a hybrid transformer-U-Net architecture that addresses this limitation through self-attention dependency-modeling in the complex spectrogram domain. U-PAST tokenizes a complex STFT representation, similarly to the magnitude spectrogram tokenization of the Audio Spectrogram Transformer (AST), applies a multi-layer transformer encoder, and reconstructs the enhanced complex spectrogram with a U-Net-style decoder. We evaluate four architectural variants with between 1.17M and 2.40M parameters on the DNS Challenge, VoiceBank-DEMAND, and LibriMix corpora under matched, acoustic mismatch, and two-dataset mismatch conditions. U-PAST attains the best SI-SDR of any evaluated model under acoustic mismatch and closely trails substantially larger convolutional and time-domain baselines by 0.26 dB to 0.63 dB SI-SDR under the remaining three conditions while achieving the strongest perceptual (DNSMOS) quality under dataset mismatch. The largest evaluated configuration, U-PAST-H (2.40M parameters), is consistently the strongest variant of the family, offering an attractive performance-to-cost trade-off at a small parameter footprint.
1 Introduction
U-PAST addresses single-channel speech enhancement by combining transformer self-attention with a complex-valued U-Net that reconstructs both magnitude and phase. Its design targets global spectro-temporal modeling while retaining hierarchical decoder upsampling and skip connections.
- 1 Introduction: Single-channel enhancement requires modeling spectral structure and temporal context because spatial information is unavailable.Environmental background noise can substantially impair speech quality, intelligibility, and downstream speech-processing systems.
- 1 Introduction: Unlike the cited transformer-only model, U-PAST processes phase and includes a convolutional decoder with hierarchical upsampling and skip connections.The comparison passage identifies noisy-phase reuse as a limitation for perceptual enhancement quality.
- 1 Introduction: U-PAST operates directly on complex-valued spectrograms and converts magnitude-phase spectra into multiple patch tokens.Self-attention lets each patch interact with all others to capture global spectro-temporal patterns.
- 1 Introduction: The architecture pairs a ViT-style transformer encoder with a complex-valued U-Net decoder containing hierarchical upsampling and lateral skip connections.The decoder reconstructs both magnitude and phase instead of reusing the noisy input phase.
2 Related Work
Prior work applies self-attention to spectrogram tokens for global audio context, while complex-valued networks process STFT information directly for speech enhancement. These approaches motivate combining transformer-style tokenization with complex-spectrogram processing and U-Net decoding.
- Audio Spectrogram Transformers: The Audio Spectrogram Transformer represents a magnitude spectrogram as patch tokens and uses self-attention to model global context and long-range dependencies.AST adapts the Vision Transformer architecture to audio without convolutional operations.
- Audio Spectrogram Transformers: AST tokenization divides a spectrogram into patches, flattens them, and projects them into an embedding space.The number of tokens depends on the patch size and whether patches overlap.
- Audio Spectrogram Transformers: A learnable [CLS] token and positional encoding augment the patch embeddings before transformer encoding.The resulting input sequence contains N + 1 tokens, with positional encodings supplying token-location information.
- Audio Spectrogram Transformers: Self-attention allows AST encoder layers to weight different spectrogram regions and capture local and global dependencies for audio classification.This provides dependency modeling beyond strictly local convolutional operations.
- Complex-Valued Speech Enhancement: Complex-valued networks operate directly on complex STFT magnitude and phase information, whereas real-valued networks treat component streams separately.Complex-valued U-Nets have been applied to single-channel speech enhancement to improve noise suppression while preserving speech intelligibility.
3 Methods
U-PAST combines an AST-based transformer encoder with a complex-valued U-Net decoder for speech enhancement. Its variants explore encoder depth, patch-based tokenization, feature projection, and a composite waveform–spectral–SI-SDR training objective.
- Encoder variants: U-PAST variants vary encoder depth and input patch size while keeping decoder depth, hidden dimension, and attention heads fixed.The family ranges from U-PAST-S with 1 encoder layer to U-PAST-H with 12 layers; all use hidden dimension 96 and 3 attention heads.
- Convolutional U-Net decoder: The decoder uses four complex-valued U-Net upsampling layers with transposed convolutions, convolutional blocks, residual connections, and CPReLU activations.This decoder follows a U-Net-style reconstruction pathway for complex-valued features.
- Tokenization: The encoder converts input features into patch embeddings by extracting, flattening, projecting, and positionally encoding patches.The algorithm stores each projected patch embedding in the token sequence after adding positional encoding.
- Feature projection: A projection bridge reshapes transformer token sequences into decoder feature maps and expands spatial resolution to support lateral skip connections.Complex transposed projection convolutions expand the base token grid by 2x, 4x, and 8x as required by decoder stages.
- Loss function: The training objective combines reconstructed waveform, multi-resolution STFT, and SI-SDR losses.The multi-resolution STFT component averages spectral convergence and log-magnitude terms over three resolutions, with λ1=1.0, λ2=1.0, and λ3=0.05.
4 Experimental Setup
The experiments use three speech corpora and evaluate models across matched and mismatch conditions, with standardized STFT inputs, optimization, metrics, and complexity reporting.
- Datasets and conditions: Three corpora—DNS Challenge, VoiceBank-DEMAND, and LibriMix—are used for analysis, with audio resampled to 16 kHz.
- Input representation: Inputs are 25,500-sample segments represented by log-magnitude and phase STFT features with shape I ∈R2×256×256.The STFT uses a 400-sample window, 100-sample shift, and 512-point FFT after removing the DC component.
- Optimization: Training uses Adam with an initial learning rate of 0.001 and a cosine schedule with 100-step linear warmup over 500 epochs.
- Evaluation: Performance is assessed with SI-SDR, PESQ, and DNSMOS, while computational cost is reported using MACs, real-time factor, and parameter counts.RTF is averaged over 20 runs after 3 warmup iterations; RTF < 1 indicates faster-than-real-time processing.
- Baselines: Complex-valued U-Net baselines use convolutional encoders with four layers and channel widths scaled between smaller and larger configurations.CUNet-S uses 32, 64, 128, and 256 channels, while CUNet-B uses 64, 128, 256, and 512.
5 Results and Discussion
U-PAST delivers a favorable complexity–performance trade-off across matched, acoustic-mismatch, and dataset-mismatch conditions. It achieves the best SI-SDR under acoustic mismatch, remains competitive elsewhere at low computational cost, and performs best in its largest configuration.
- Complexity: 1.17M parameters and 4.39 GMAC: U-PAST-S is the smallest and least costly configuration, with only mild complexity growth across the family.U-PAST uses substantially fewer MACs than CUNet-B, MP-SENet, and TF-GridNet, though its RTF remains higher than purely convolutional models.
- Matched condition: 15.69 dB SI-SDR: U-PAST-H trails TF-GridNet by 2.44 dB in the matched condition but achieves the highest PESQ-wb at 3.41.It also surpasses CleanUNet and matches CUNet-S at a comparable parameter budget.
- Acoustic mismatch: 9.76 dB SI-SDR: U-PAST-H achieves the best result under acoustic mismatch, while every U-PAST variant exceeds the unprocessed baseline.U-PAST-S also achieves the best PESQ-wb and PESQ-nb in this condition.
- Dataset mismatch: 16.72 dB SI-SDR: U-PAST-H trails TF-GridNet by 1.62 dB on VoiceBank-DEMAND while using 3.5× fewer parameters than CUNet-B.U-PAST variants remain competitive on perceptual scores, including the best non-TF-GridNet MOS-sig and MOS-ovr results from U-PAST-B.
- Dataset mismatch: 10.30 dB SI-SDR: U-PAST-H trails TF-GridNet by 1.73 dB on LibriMix and remains ahead of CUNet-S and U-PAST-L.TF-GridNet leads most perceptual sub-scores in this condition.
- Encoder size: 2.40M parameters: U-PAST-H is consistently the strongest family variant across all four test conditions.Increasing encoder capacity produces near-monotonic SI-SDR gains in three conditions, with a small dip for U-PAST-L under acoustic mismatch before U-PAST-H recovers the best result.
- Overall results: U-PAST achieves the best SI-SDR of any evaluated model under acoustic mismatch, while trailing TF-GridNet by 1.62–2.44 dB in the other three conditions.The family requires 18–19× fewer MACs and has more than an order of magnitude lower RTF than TF-GridNet.
6 Conclusion
U-PAST combines transformer self-attention with complex-valued U-Net decoding to provide competitive enhancement performance at a small parameter footprint. Its accuracy-to-compute trade-off is not yet matched by wall-clock efficiency, motivating inference-side optimization.
- 1.17M–2.40M parameters keep U-PAST competitive with baselines up to 39× larger across matched and mismatch conditions.CleanUNet has 46.07M parameters, while TF-GridNet has a comparable 1.34M-parameter count.
- 9.76 dB SI-SDR makes U-PAST-H the best evaluated model under acoustic mismatch.U-PAST-H is the largest evaluated configuration.
- U-PAST-H is the strongest family configuration in every evaluated condition, with encoder-depth increases producing near-monotonic SI-SDR gains in three conditions.The comparison spans U-PAST-S through U-PAST-H, with encoder depth increasing from 1 to 12 layers.
- U-PAST achieves favorable parameter and compute efficiency, but its A100 RTF remains higher than purely convolutional CleanUNet and CUNet.U-PAST’s RTF is 0.0126–0.0151, compared with 0.0058 for CleanUNet and 0.0081–0.0115 for CUNet.
- Further work targets smaller patch sizes and inference-time efficiency to translate U-PAST’s small footprint into a wall-clock advantage.