PaperlayerView exact revision on arXiv

Source-linked AI summary

GrainSpeech: Less Context, More Detail for Compact Speech Synthesis

Zitao Liang, Chang Gao

arXiv:2609.18856v1eess.AScs.AIcs.SDeess.SP

TL;DR

Compact acoustic models must balance synthesis quality with tight capacity limits, and encoder-context effects and Mel-specific supervision remain insufficiently isolated. GrainSpeech combines a fixed-receptive-field convolutional encoder with Mel-adapted gradient-variance supervision; at 264.8K parameters, it improves UTMOS over equal-budget ES-Tiny and generates Mel in real time on an MCU.

  • Problem

    Compact acoustic models face a quality-capacity trade-off, while encoder-context effects and Mel-specific supervision remain insufficiently isolated.

    Source: §1 · p. 1

  • Method

    GrainSpeech combines a fixed-receptive-field convolutional encoder with Mel-adapted gradient-variance supervision.

    Source: §1 · p. 1, §2.2 · p. 2

  • Results

    264.8K-parameter GrainSpeech raises UTMOS from 3.591 to 4.086, is statistically comparable to MixerTTS with 75.7× fewer parameters, and generates Mel at 17.9× real time on an STM32H747XI MCU.

    Source: §5 · p. 4

  • Takeaways & Limitations

    Under the evaluated setting, self-attention beyond 15 phonemes provides no consistent benefit, and Mel-GVar avoids the quality degradation observed with direct image-domain transfer.

    Source: §3.3 · p. 3, §5 · p. 4

    Evaluation relies on automatic metrics on LJSpeech, and external checkpoints use native settings rather than controlled comparisons.

    Source: §4 · p. 4

Abstract

from arXiv · show

Compact acoustic models face a challenging quality-capacity trade-off. We investigate two factors in this regime: encoder context and Mel-spectrogram supervision. A receptive-field-scaling study shows that expanding self-attention beyond 15 phonemes provides no consistent gains in pitch, energy, or duration prediction. Guided by this finding, we introduce a fixed-receptive-field convolutional encoder that reduces the respective prediction errors by 36.0%, 17.3%, and 3.4%. We further show that directly transferring image-domain gradient-variance supervision restores fine-scale variation but degrades predicted quality, motivating a Mel-specific formulation with axis-specific gradients, overlapping local statistics, and log-domain variance matching. GrainSpeech contains only 264.8K parameters and achieves 17.9x real-time Mel generation on a microcontroller (MCU), while attaining UTMOS scores comparable to substantially larger models with less than 1.5% of their parameters. Source code and demos are available at https://github.com/lab-emi/GrainSpeech.

1. INTRODUCTION

Compact acoustic models must balance synthesis quality against tight capacity limits, while encoder-context effects and Mel-specific supervision remain insufficiently isolated. GrainSpeech addresses these challenges with a fixed-receptive-field encoder and Mel-adapted gradient-variance supervision.

  • Prior encoder comparisons confounded architecture with accessible context, while pointwise L1 risks oversmoothing and image-domain gradient variance was designed for spatial structure.Source: §1 · p. 1
  • 36.0%, 17.3%, and 3.4% lower pitch, energy, and duration errors, respectively, with a parameter-matched fixed-receptive-field convolutional encoder.Self-attention beyond 15 phonemes provides no consistent gain.Source: §1 · p. 1
  • Mel-GVar adapts gradient-variance supervision with axis-specific gradients, overlapping local statistics, and log-domain matching; direct transfer yields UTMOS 2.769, versus 4.086 with Mel-GVar.Source: §1 · p. 1
  • 264.8K-parameter GrainSpeech gains 0.496 UTMOS over equal-budget ES-Tiny, matches MixerTTS statistically with 75.7× fewer parameters, and generates Mel at 17.9× real time on an STM32H747XI MCU.Source: §1 · p. 1

2. PROPOSED METHOD

The proposed method combines a fixed-receptive-field convolutional encoder with an anti-oversmoothing Mel objective. The encoder controls phoneme context, while the objective adds local-variation supervision to the Mel loss.

2.1. System Overview

  • The encoder predicts pitch, energy, and duration from phonemes; duration-based upsampling precedes dilated temporal mixing and a channel-mixing bottleneck that generates Mel.Training uses the anti-oversmoothing Mel objective.Source: §2.1 · pp. 1–2

2.2. Fixed-Receptive-Field Encoder

  • Attention context W is controlled with symmetric masks on the original phoneme sequence; stacked residual convolutional blocks use dilation schedules to realize matched W.Downstream predictors expand context equally across variants, enabling matched-context comparisons.Source: §2.2 · p. 2

2.3. Anti-Oversmoothing Mel Loss

  • Mel-GVar uses time- and frequency-axis first differences, overlapping 11×5 local variance windows, and log-domain L1 matching; the Mel objective combines L1, SSIM, and weighted Mel-GVar.The local statistics use unit-stride windows and exclude padded elements.Source: §2.3 · p. 2

3. EXPERIMENTS

Experiments compare encoder context and architecture, Mel-spectrogram objectives, and final model quality and deployment. GrainSpeech combines a fixed-context convolutional encoder with Mel-GVar and is evaluated on LJSpeech, including MCU Mel generation.

3.1. Experimental Setup

  • Experiments use LJSpeech with 12,588 training utterances and 128 evaluation utterances; all variants share a training seed and are evaluated after 5,000 epochs.The 512 remaining utterances are split into 384 validation and 128 evaluation utterances.Source: §3.1 · p. 2

3.2. Encoder Receptive Field and Architecture Study

  • At W = 15, the convolutional encoder reduces pitch, energy, and duration MSE by 36.0%, 17.3%, and 3.4%, respectively, while attention gains no consistently beyond W = 15.Parameter-matched encoders are compared with downstream components and the Mel objective unchanged; W = 15 is selected using validation acoustic-feature errors.Source: §3.2 · p. 3

3.3. Oversmoothing Analysis and Loss Ablation

  • Adding SSIM improves UTMOS from 3.798 to 3.923, but corner-region attenuation remains visible in the normalized 2D Fourier spectrum.L1 predictions show smoother Mel patterns and attenuated corner energy relative to ground truth.Source: §3.3 · p. 3, Figure 3 · p. 4
  • Mel-GVar yields the highest UTMOS among evaluated objectives at 4.086, whereas direct image-domain GVar lowers UTMOS to 2.769 despite restoring fine-scale variation.Mel-GVar also improves ES-Tiny UTMOS from 3.591 to 3.945, with similar WER (3.08% vs. 3.13%).Source: §3.3 · p. 3

3.4. Overall Comparison

  • GrainSpeech improves UTMOS by 0.496 over ES-Tiny at the same parameter budget and is statistically indistinguishable from MixerTTS with 75.7× fewer parameters; it generates Mel at 17.9× real time on an STM32H747XI.The MCU deployment uses 483.19 KiB peak SRAM; ES-Small exceeds available SRAM, and quantization reduces UTMOS by 0.036 for GrainSpeech.Source: §3.4 · pp. 3–4, Table 1 · p. 4

4. DISCUSSION

The discussion identifies limits on interpreting the evaluation and on generalizing the study's findings. It also notes that Mel-GVar's local-statistics objective and MCU evaluation leave some questions outside the study's scope.

4. DISCUSSION

  • UTMOS gains rely on automatic metrics on LJSpeech; listening tests and broader datasets are needed for confirmation, and external checkpoints are reference points rather than controlled comparisons.ES-Tiny rows share the authors' pipeline, unlike external checkpoints with native data splits, training settings, and frontends.Source: §4 · p. 4
  • The context finding is specific to the evaluated architecture and training setting and does not establish a universal receptive-field requirement.Source: §4 · p. 4
  • Mel-GVar constrains local variation statistics rather than exact detail locations; its three adaptations are evaluated jointly, and end-to-end deployment including vocoding is outside scope.Recovered details need not match the reference point by point, and Table 2 covers Mel generation only.Source: §4 · p. 4

5. CONCLUSION

GrainSpeech combines compact acoustic modeling with a fixed-context convolutional encoder and Mel-GVar supervision. In the reported setting, it improves UTMOS at the ES-Tiny budget, remains statistically comparable to MixerTTS with far fewer parameters, and generates Mel in real time on an MCU.

5. CONCLUSION

  • At the ES-Tiny budget, GrainSpeech raises UTMOS from 3.591 to 4.086 and is statistically comparable to MixerTTS with 75.7× fewer parameters.The model contains 264.8K parameters.Source: §5 · p. 4
  • 17.9× real-time Mel generation uses 483 KiB of SRAM on an STM32H747XI MCU.This is reported under the paper’s experimental setting.Source: §5 · p. 4
  • Beyond 15 phonemes, self-attention provides no consistent benefit, while the matched-context convolutional encoder improves acoustic-feature prediction.Mel-GVar restores fine-grained variation while avoiding the quality degradation observed with direct transfer of the image-domain objective.Source: §5 · p. 4

6. COMPLIANCE WITH ETHICAL STANDARDS

The study uses publicly available LJSpeech and reports no recruitment, new recordings, human listening tests, or animal experiments.

6. COMPLIANCE WITH ETHICAL STANDARDS

  • The study uses the publicly available LJSpeech dataset.No new recordings were collected.Source: §6 · p. 5
  • No participants were recruited, and no human listening tests were conducted.Source: §6 · p. 5
  • No animal experiments were conducted.Source: §6 · p. 5
Loading 2609.18856v1…