PaperlayerView exact revision on arXiv

Source-linked AI summary

Monotonicity-Guided Semantic Alignment for Zero-shot Multispeaker Image-to-Speech Synthesis

Lijun Wang, Yixian Lu, Shogo Okada

arXiv:2609.38440v1eess.AS

TL;DR

Direct Img2Sp lacks a predefined alignment between image content and ordered speech, and prior methods do not demonstrate reference-conditioned synthesis for unseen speakers. MGSA aligns visual-semantic memory to speech-unit positions and generates units with blockwise masked diffusion. It achieves competitive captioning performance against single-speaker baselines while supporting zero-shot synthesis for unseen speakers.

  • Problem

    Prior Img2Sp methods do not demonstrate reference-conditioned synthesis for unseen speakers, while images lack a predefined monotonic mapping to ordered speech units.

    Source: §1 · p. 1

  • Method

    MGSA uses semantic speech units and reference speech for speaker conditioning, aligns visual-semantic memory to speech-unit positions with a soft monotonic prior, and generates units through blockwise masked diffusion.

    Source: §1 · pp. 1–2, §2 · p. 2

  • Results

    MGSA achieves competitive performance against single-speaker baselines while supporting zero-shot multispeaker synthesis.

    Source: §1 · p. 2, §5 · p. 4

  • Takeaways & Limitations

    With unseen LibriTTS-R speakers as references, MGSA maintains captioning performance comparable to seen-speaker settings and obtains higher DistillMOS and UTMOS scores.

    Source: §4.2 · p. 4

    Captioning comparisons with single-speaker baselines do not test their performance under the additional unseen-speaker generalization challenge.

    Source: §4.1 · p. 4

Abstract

from arXiv · show

Direct image-to-speech (Img2Sp) poses an alignment challenge in mapping visual content to ordered speech sequences, as images permit multiple spoken descriptions and lack monotonic correspondence with speech sequences. We propose Monotonicity-Guided Semantic Alignment (MGSA), to the best of our knowledge, the first framework for zero-shot multispeaker Img2Sp synthesis. We use semantic speech units to provide shared content targets across speakers with reference speech for speaker conditioning. A query aligner maps semantic memory learned from visual content to speech unit positions via a soft monotonic prior, which yields position-specific conditioning states. A blockwise masked diffusion generator is employed for the speech unit generation conditioning on these states. Experiments on Flickr8k-Audio show competitive captioning performance against single-speaker baselines, while evaluation with LibriTTS-R references supports zero-shot synthesis for unseen speakers. Ablations validate the effectiveness of aligner and block diffusion. Audio samples are available at https://alizeded.github.io/mgsa-demo.

1. INTRODUCTION

Direct image-to-speech synthesis must map visual content to ordered speech despite images allowing multiple descriptions and lacking a predefined alignment to speech positions. MGSA addresses this challenge for zero-shot multispeaker synthesis by aligning visual semantics to speech units and generating them with blockwise masked diffusion.

1. INTRODUCTION

  • Existing Img2Sp methods do not demonstrate reference-conditioned synthesis for unseen speakers, motivating zero-shot multispeaker generation.Prior systems use single-speaker synthesized speech or a vocoder trained on LJSpeech.Source: §1 · p. 1
  • Images permit multiple descriptions and lack predefined monotonic correspondence to speech-unit positions, complicating visual-to-speech alignment.Speaking-rate and acoustic variation across unseen speakers adds difficulty, while reference speech and teacher-forced prefixes may let the model rely less on image conditions.Source: §1 · p. 1
  • MGSA uses semantic speech units as shared content targets across speakers and reference speech for speaker conditioning.The approach addresses concern that regressing continuous features may prevent focus on image content.Source: §1 · p. 1
  • Blockwise masked diffusion generates units block by block until termination, combining flexible-length generation with inference that limits local-prefix access.Unlike token-by-token autoregressive decoding, it does not require a predefined duration; masking may reduce reliance on prefix continuation over image conditioning.Source: §1 · p. 1
  • A monotonic query aligner maps learned visual-semantic memory to position-specific speech-unit conditions without access to speech-unit prefixes.The soft monotonic prior acts on learned memory, not the spatial order of image features.Source: §1 · pp. 1–2
  • The authors present MGSA as, to their knowledge, the first framework for zero-shot multispeaker Img2Sp and report competitive performance against single-speaker baselines.They also report support for unseen speakers with excellent speech quality.Source: §1 · p. 2

2. PROPOSED METHOD

MGSA encodes image content and speaker style into semantic memory, aligns that memory to speech-unit positions, and generates units block by block. Unit and occupancy objectives train the aligner, while masked prediction trains the generator.

2. PROPOSED METHOD

  • Given an image, reference speech, and optional prefix, MGSA generates speech units that a reference-conditioned unit-to-speech model converts into a waveform.The reference supplies speaker conditioning and the image supplies content conditioning.Source: §2 · p. 2
  • The semantic encoder maps visual features and speaker conditioning into fixed-capacity indexed memory, which is mean-pooled for global semantic conditioning.A frozen visual encoder first extracts image representations.Source: §2 · p. 2
  • The monotonic query aligner attends to memory with a soft monotonic prior to produce position-specific states for blockwise unit generation.Its shared queries use positional information, and the aligner receives image and speaker conditions but not speech-unit prefixes.Source: §2 · p. 2
  • The generator predicts each block conditioned on completed blocks, aligner states, global semantic conditioning, and speaker style.The sequence is padded with EOS to a block boundary before block generation.Source: §2 · p. 2
  • During training, masked diffusion replaces a subset of active-block positions with MASK tokens and optimizes cross-entropy on those positions.At inference, each block starts fully masked and is iteratively refined; candidates are ranked by confidence perturbed with annealed Gumbel noise, and EOS is suppressed before the minimum length.Source: §2 · pp. 2–3
  • Unit prediction and occupancy prediction supervise aligner states, with occupancy supervision covering all L + B states and used only during training.The aligner loss combines unit and occupancy losses, and the total MGSA loss adds the generator's masked-prediction objective.Source: §2 · p. 3

3. MODEL ARCHITECTURE

The architecture encodes visual features with speaker conditioning into semantic memory, then uses a monotonic query aligner to form position-specific states for a blockwise diffusion generator.

3. MODEL ARCHITECTURE

  • A frozen BLIP2 Q-former extracts 32 visual tokens, and the semantic encoder maps visual features with speaker conditioning into a 320×768 memory.Reference speech is encoded as an ECAPA-TDNN speaker vector and four FACodec timbre and prosody vectors; the speaker token is appended to memory for MQA.Source: §3 · p. 3
  • The two-layer monotonic query aligner uses positional encoding and a soft monotonic bias to attend from speech-unit positions to semantic memory.Its block-causal self-attention restricts each query to its own and preceding blocks; the bias encourages attention near the nominal memory slot for each position.Source: §3 · p. 3
  • The block masked diffusion generator is an eight-layer, 512-dimensional Diffusion Transformer whose input combines a token embedding with projected alignment states.The input equation specifies the token as a unit, mask, or EOS token.Source: §3 · p. 3
  • Classifier-free guidance is applied only to image conditioning, with the null branch replacing visual features with zeros while retaining speaker conditioning.Source: §3 · p. 3

4. EXPERIMENTS

Experiments evaluate MGSA on captioning, speech quality, and speaker similarity, including references from unseen speakers. MGSA compares favorably with single-speaker baselines, and ablations support the alignment and blockwise diffusion designs.

4.1. Experimental Setup

  • Training uses Flickr8k-Audio and SpokenCOCO; evaluation includes 38 unseen LibriTTS-R speakers, three reference protocols, and captioning, naturalness, and speaker-similarity metrics.Source: §4.1 · pp. 3–4

4.2. Experimental Results

  • Removing MQA while retaining semantic content and speaker conditioning, and testing mismatched images, supports MQA’s effectiveness and confirms image conditioning.Source: §4.2 · p. 4
  • The autoregressive variant performs worse than blockwise masked diffusion and degrades further with mismatched images, supporting the blockwise design.Source: §4.2 · p. 4
  • With unseen LibriTTS-R references, MGSA has captioning performance comparable to seen-speaker settings, higher DistillMOS and UTMOS, and speaker similarity close to same-speaker natural recordings.Generated-to-reference similarity is well above that of different-speaker natural recordings.Source: §4.2 · p. 4, Table 2 · p. 4, Table 3 · p. 4
  • MGSA outperforms SAS, SAT, SAT-FT, and PixVoxLM; under the unseen protocol it also exceeds one Im2Sp variant, while GiT-initialized Im2Sp has higher captioning scores.MGSA supports zero-shot multispeaker synthesis for unseen speakers.Source: §4.2 · p. 4, Table 2 · p. 4

5. CONCLUSION & LIMITATION

MGSA combines monotonicity-guided alignment and block diffusion for zero-shot multispeaker synthesis. Experiments show competitive performance against single-speaker baselines, while captioning performance remains a practical limitation.

5. CONCLUSION & LIMITATION

  • MGSA combines monotonicity-guided alignment and block diffusion in a framework for zero-shot multispeaker synthesis.Source: §5 · p. 4
  • Experiments show competitive performance against single-speaker baselines while supporting zero-shot multispeaker synthesis.Source: §5 · p. 4
  • Captioning performance remains a limitation for practical application, motivating improvements in visual grounding and I2U content accuracy.Source: §5 · p. 4

Related papers

Loading 2609.38440v1…