Source-linked AI summary
Configurable-Bandwidth Time-Frequency Modeling for Efficient Full-Band Speech Enhancement Across Sampling Rates
Ui-Hyeop Shin, Wooseok Kim, Hyung-Min Park
TL;DR
Speech enhancement across sampling rates is costly for time-frequency models, while existing practical frequency allocations are fixed to a design and rate. TF-Refiner uses configurable-bandwidth deep analysis with full-band decoding and filtering; its universal model performs well across rates, and random-cutoff training supports inference-time cost–quality selection.
Problem
Time-frequency model cost grows with frequency-bin count, while practical full-band designs allocate analysis effort in fixed structures tied to a sampling rate.
Source: §1 · p. 1Method
TF-Refiner deeply analyzes frequencies below a configurable cutoff and uses a shallow full-band decoder with input-dependent high-band queries to predict local filters for the original noisy STFT.
Source: §1 · p. 1, §2 · p. 2, §2.3 · p. 2Results
The universal model trained at 16 and 48 kHz achieves the best PESQ, STOI, and LSD at every evaluated rate, including unseen rates, while 16-kHz training retains the highest SI-SDR at 44.1 and 48 kHz.
Source: §4.1 · pp. 3–4Takeaways & Limitations
Random-cutoff training lets the analysis cutoff select a cost–quality operating point at inference without retraining or changing the full-band output.
Source: §1 · p. 2, §5 · p. 4Table 1 does not isolate whether the universal model’s advantage comes from data diversity or joint exposure to multiple sampling rates.
Source: §4.1 · p. 4
Abstract
from arXiv · showhide
Speech enhancement systems are often developed for a fixed sampling rate, while time-frequency models become more expensive as the number of frequency bins increases. We propose TF-Refiner, a sampling-frequency-independent model that decouples the deep analysis bandwidth from the full-band input and output. A deep encoder processes the band below a configurable cutoff, while a shallow decoder combines the encoded features with input-dependent high-band queries and predicts local complex filters applied to the original noisy STFT. A single parameter set trained at 16 and 48 kHz is evaluated at various sampling rates. On VoiceBank+DEMAND, the universal model outperforms the rate-specific counterparts in PESQ, STOI, and log-spectral distance across the evaluated rates, including rates unseen in training. Random-cutoff training enables inference-time selection of cost-quality operating points without retraining or changing the output bandwidth. These results support configurable analysis bandwidth as a practical design choice for multi-rate full-band enhancement.
1. INTRODUCTION
The paper addresses the cost of applying time-frequency models across sampling rates and proposes TF-Refiner, which confines deep analysis to a configurable physical-frequency band while retaining full-band input and output. A universal model performs competitively across rates, and random-cutoff training supports inference-time cost–quality adjustment.
1. INTRODUCTION
- Existing sampling-frequency-independent enhancement systems are computationally demanding, and time-frequency dual-path model cost grows almost proportionally with the number of frequency bins.The architecture becomes substantially more expensive at 48 than at 16 kHz.Source: §1 · p. 1
- Practical full-band systems use fixed frequency structures, tying their allocation of analysis effort to a design-time choice and one sampling rate.Examples include perceptual-band envelopes, low-band complex filtering, and fixed sub-band networks.Source: §1 · p. 1
- TF-Refiner deeply analyzes frequencies below configurable fc, uses the complementary observed band as input-dependent queries, and predicts local complex filters for the original noisy STFT.Its parameters do not depend on the number of frequency bins, allowing one parameter set to serve any sampling rate.Source: §1 · p. 1, Figure 1 · p. 1, Figure 2 · p. 2
- A universal model trained at 16 and 48 kHz matches or exceeds rate-specific training in perceptual quality and generalizes to unseen rates; random-cutoff training lets fc set inference-time cost–quality trade-offs without retraining.Source: §1 · pp. 1–2
2. TF-REFINER
TF-Refiner keeps its analysis cutoff at the same physical frequency across sampling rates, then uses asymmetric encoding and decoding to refine the full observed spectrum. Its Filter Head applies locally predicted complex filters to the original noisy STFT.
2. TF-REFINER
- A physical-frequency cutoff fc sets the deep analysis band; for a given cutoff, the sampling rate affects only the size of the complementary band.The SFI-STFT uses 25-Hz bin spacing at each rate, and the band partition is drawn at fc.Source: §2 · p. 2, §2.1 · p. 2
2.1. SFI representation and band partition
- Randomly sampling fc during training lets one parameter set serve different analysis cutoffs at inference.The SFI representation uses a 40-ms window and 20-ms hop at every sampling rate.Source: §2.1 · p. 2
2.2. Asymmetric encoder–decoder
- The deep TF-Encoder processes the low band, while the shallow TF-Decoder uses the complementary band as queries, attends to encoder features, and relates both bands with full-band attention.Band Merge concatenates the high-band queries with encoder features; low-band positions bypass high-band cross-attention.Source: §2.2 · p. 2
2.3. Deep filtering
- The Filter Head predicts local complex filters from decoder features and applies them to neighboring coefficients of the original noisy STFT.Compressed features determine the filter, while the samples being combined are the uncompressed observations.Source: §2.3 · p. 2
3. EXPERIMENTAL SETUP
The evaluation uses VoiceBank+DEMAND test speech at multiple input rates and reports perceptual, intelligibility, signal-to-distortion, and spectral-distance metrics. Training alternates 16-kHz and 48-kHz data streams and samples analysis cutoffs.
3.1. Datasets and evaluation metrics
- The evaluation uses 824 VoiceBank+DEMAND test utterances at 16 and 48 kHz, with the 48-kHz training pool built from VCTK speech and DNS full-band noise.The 16-kHz training set contains 10,413 utterances, and the 48-kHz stream excludes VCTK utterances overlapping the VBD validation set.Source: §3.1 · pp. 2–3
- Figure 5 compares inference cutoffs for 48-kHz and 16-kHz inputs, distinguishing fixed-cutoff controls from the randomly trained universal model.Its caption marks the default 5-kHz cutoff in red and identifies the training cutoff for fixed-cutoff models with circles.Source: Figure 5 · p. 3
- The study reports wideband PESQ, STOI, SI-SDR, and full-band LSD, with LSD above 8 kHz also reported where defined.The multi-rate test pairs are evaluated at 16, 24, 32, 44.1, and 48 kHz; MACs are measured per second of input.Source: §3.1 · p. 3
3.2. Training and model configuration
- The universal model alternates 16-kHz VBD and 48-kHz VCTK+DNS batches and samples cutoffs from rate-specific sets during training.A single checkpoint per model is used across input rates and inference cutoffs; the default cutoff is 5 kHz.Source: §3.2 · p. 3
4. RESULTS
The universal model performs best on PESQ, STOI, and LSD across evaluated rates, while random-cutoff training offers inference-time cost–quality choices. Published-system comparisons and ablations show how the variants trade quality, cost, and high-band fidelity.
4.3. Comparison with published systems
- At 16 kHz, TF-Refiner-B (16k) reaches PESQ 3.35 versus FastEnhancer-M’s 3.24; at 48 kHz, TF-Refiner-B (48k) reaches 3.35 versus DeepFilterNet2’s 3.08.The universal TF-Refiner-B reaches PESQ 3.46 and 3.47 at 16 and 48 kHz, respectively.Source: Table 2 · p. 4, §4.3 · p. 4
4.1. Universal enhancement across sampling rates
- The universal model achieves the best PESQ, STOI, and LSD at every rate, and the best SI-SDR through 32 kHz; rate-specific models also transfer to unseen rates.At 44.1 and 48 kHz, the 16-kHz model retains the highest SI-SDR; at a 5-kHz cutoff, cost grows about 2.0× from 16 to 48 kHz while frequency bins triple.Source: §4.1 · pp. 3–4
- VBD and VCTK training streams do not provide disjoint clean-speech corpora, and the evaluation does not isolate data diversity from joint sampling-rate exposure.The 48-kHz-only model uses the larger VCTK+DNS pool but remains below the universal model at its native rate.Source: §4.1 · p. 4
4.2. Configurable computation
- Random-cutoff training keeps PESQ and LSD near the best fixed-cutoff model over 2–8 kHz, while fixed-cutoff models retain an advantage at the extremes.For 16-kHz input, the near-best range is 2–7 kHz.Source: §4.2 · p. 4
- Lowering the cutoff from 5 to 3 kHz reduces cost by 29% at 16 kHz and 16% at 48 kHz, with a small PESQ decrease.At 48 kHz, the fixed 16-kHz-cutoff model costs about twice as much as the default universal model without higher PESQ.Source: §4.2 · p. 4
4.3. Comparison with published systems
- The no-look-ahead variants use 40-ms algorithmic latency; S† maintains comparable quality to B† at substantially lower cost, while T† trades some quality for further savings.TF-Refiner-B† also outperforms the published systems in PESQ at both rates.Source: §4.3 · p. 4
4.4. Ablation study
- Removing high-band MHCA or Band Merge changes below-8-kHz PESQ and STOI by at most 0.04 but worsens SI-SDR and LSD above 8 kHz; direct complex mapping degrades every metric.The two module removals raise LSD>8k from 0.730 to 0.794 and 0.773, respectively; direct mapping lowers SI-SDR by 1.8 dB and raises LSD by 0.07.Source: §4.4 · p. 4, Table 4 · p. 4
5. CONCLUSION
TF-Refiner separates configurable deep analysis from full-band input and output. A single model covers 16- to 48-kHz inputs, including unseen rates, while the cutoff selects an inference-time cost–quality point without retraining.
5. CONCLUSION
- TF-Refiner confines deep analysis to a configurable band while a shallow decoder refines the observed remainder and filters the original noisy STFT.One universal model enhances 16- to 48-kHz inputs, including unseen rates, and the cutoff selects a cost–quality point without changing the architecture or retraining.Source: §5 · p. 4