Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS

• End-to-end neural TTS commonly maps characters or phonemes to acoustic features and uses a neural vocoder to generate high-quality waveforms.Source: §1 · p. 1
• Out-of-domain, long, complicated, or abnormal inputs can trigger skipping, repeating, and attention collapse, limiting broader neural TTS use.Source: §1 · p. 1
• The paper models TTS alignments as a surjective mapping from output frames to input tokens, requiring locality, monotonicity, and completeness.These criteria respectively avoid attention collapse, prevent repeating, and avert skipping.Source: §1 · p. 1
• Stepwise monotonic attention adds completeness constraints to monotonic attention, producing stable behavior on singular inputs while preserving in-domain naturalness comparable to state-of-the-art models.Source: §1 · p. 1

2.1. Attention-based neural TTS

• At each decoder step, attention scores encoder memory entries, normalizes them into an alignment vector, forms a context vector, and generates acoustic outputs autoregressively.Source: §2.1 · p. 1
• Prior attention variants target locality, monotonicity, and completeness through explicit alignment modeling, penalties, location sensitivity, or forward variables.Source: §2.1 · p. 2

2.2. Monotonic attention

• Monotonic attention scans memory entries forward from the previous focus and selects one using Bernoulli-based stopping probabilities.Source: §2.2 · p. 2
• This hard selection guarantees one focused memory entry per step with no backward movement, while retaining linear time complexity and online-input support.Source: §2.2 · p. 2
• Because hard attention is nondifferentiable, monotonic attention uses recursively or parallelly computed soft alignment distributions during training.The expected context vector replaces direct memory selection, with normalization, noise, and bias tricks supporting optimization and movement control.Source: §2.2 · p. 2

3. Stepwise monotonic attention

• Stepwise monotonic attention limits each hard-alignment move to at most one input position, adding completeness to monotonic attention’s locality and monotonicity.Source: §3 · p. 2
• During inference, the mechanism samples whether to stay or advance one step, and its distribution can be computed recursively or efficiently with zero-padding.Source: §3 · p. 2
• Training uses expected context vectors, while linear time complexity, online decoding, noise, trainable bias, and energy normalization remain available.Source: §3 · p. 2
• The method shares mathematical form with part of forward attention but directly models TTS-specific alignment structure more simply.Source: §3 · p. 2
• Soft training creates context mismatch because the decoder sees a distribution of memory entries rather than the single entry selected during inference.Source: §3 · p. 2
• Soft inference may avoid context mismatch but sacrifices strict locality, monotonicity, linear complexity, and online decoding guarantees.Source: §3 · p. 3
• The method is proposed and evaluated for scripts containing phonemes and punctuations, whose punctuation tokens do not directly map to speech segments.Source: §3 · p. 3

4.1. Experiment settings

• The experiments used Tacotron2 variants trained on 20.8 hours of professional female speech with manually checked phoneme inputs and evaluated 80 in-domain, 80 News, and 292 difficult utterances.Models were trained for 120k steps with the same general hyperparameters and synthesized waveforms using a WaveNet vocoder.Source: §4.1 · p. 3

4.2. Naturalness evaluation

• Naturalness comparisons used crowdsourced preference tests covering all 160 General Sentence and News utterances, with 20 judgments per utterance.Results were reported with p-values in Tables 1 and 2.Source: §4.2 · p. 3, Table 1 · p. 3, Table 2 · p. 3
• SMA models achieved naturalness at least on par with the baseline and other peers, unlike MA models, which regressed especially on complicated News inputs.MA outputs showed skipping, unnatural prosody, and unstable speech speed; SMA avoided a general naturalness penalty.Source: §4.2 · p. 3

4.3 Intelligibility evaluation

• MA hard generally avoided attention collapse and repeating but often violated completeness, skipping substantial speech, whereas FA reduced but did not eliminate skipping or repeating.The analysis links these error patterns to the distinct alignment constraints of the attention mechanisms.Source: §4.3 · p. 4
• SMA models delivered the best intelligibility robustness in most cases, with near-to-perfect correctness and only few minor misalignments.The evaluation used case-level unintelligible rate and word errors on long, complex, or semantically abnormal utterances.Source: §4.1 · p. 3, §4.3 · p. 4, Table 3 · p. 4
• Soft inference hurt MA robustness because uncertain alignments averaged incompatible hard paths and could gradually produce attention collapse and unintelligible speech.SMA soft retained highly discrete alignments on the same inputs and evaded context mismatch without harming robustness.Source: §4.3 · p. 4

• Stepwise monotonic attention greatly improves robustness of attention-based neural TTS models under out-of-domain scenarios.The method is also proposed for possible application to audio, handwriting, and inflection generation.Source: §5 · p. 4