PaperlayerView exact revision on arXiv

Source-linked AI summary

ImageTTS: Repurposing Pretrained Image Diffusion Transformers for Speech Synthesis

Hanchen Pei, Shujie Liu, Yayue Deng, Fangyun Wei, Yanqing Liu, Jianwei Yu, Gongping Huang

arXiv:2610.34807v1cs.SDcs.MM

TL;DR

ImageTTS addresses whether image-model representations and generative priors can transfer to speech, which demands both linguistic fidelity and speaker identity preservation. It repurposes FLUX.2 for speech and separates transfer through the image VAE from transfer through pretrained DiT initialization. At scale, it reports results on both zero-shot and instruction-guided TTS.

  • Problem

    Whether pretrained image-model representations and generative priors can transfer beyond vision to speech remains unresolved, despite speech requiring linguistic fidelity and preservation of speaker identity.

    Source: §1 · pp. 1–2

  • Method

    ImageTTS encodes pseudo-RGB mel-spectrograms with a frozen image VAE and adapts a pretrained image DiT, separating representation transfer from generative-prior transfer.

    Source: §1 · p. 2

  • Results

    At scale, ImageTTS achieves 1.12% WER on Seed-TTS test-en and a Style-ACC of 0.66 on CapSpeech across zero-shot and instruction-guided TTS.

    Source: §5 · pp. 9–10

  • Takeaways & Limitations

    The results suggest pretrained image-generation models can transfer beyond architectural reuse, supplying useful latent representations and generative priors for speech synthesis.

    Source: §5 · p. 10

Abstract

from arXiv · show

Pretrained image-generation models have been successfully repurposed for various downstream visual tasks. Whether their learned representations and generative priors can transfer beyond vision to other modalities remains largely unresolved, particularly for speech synthesis, which requires linguistic fidelity and preservation of speaker identity from a short reference. We present ImageTTS, which repurposes FLUX.2 for speech synthesis by encoding pseudo-RGB log-mel spectrograms with its frozen image VAE and adapting only its pretrained Diffusion Transformer (DiT) using conditional latent flow matching. This formulation disentangles two forms of vision-to-speech transfer: representation transfer through the image VAE and generative-prior transfer through DiT initialization. The frozen image VAE largely preserves intelligibility and speaker identity with 0.99 STOI and 0.95 speaker similarity, while image-pretrained DiT initialization accelerates convergence and improves linguistic accuracy, reducing WER from 4.2\% to 2.7\% at 100k matched updates relative to random initialization. The same framework supports both zero-shot and instruction-guided TTS and, when scaled to 100K hours of speech data, achieves 1.12\% WER on Seed-TTS test-en and a Style-ACC of 0.66 on CapSpeech, outperforming speech-native baselines trained on comparable data across both tasks. These results suggest that pretrained image models can transfer beyond architectural reuse, supplying useful latent representations and generative priors for intelligible and controllable speech synthesis.

1 INTRODUCTION

ImageTTS studies whether pretrained image representations and generative priors can transfer to speech synthesis, where linguistic fidelity and speaker identity must both be preserved. It adapts FLUX.2 for speech and evaluates transfer in the latent representation, DiT initialization, and scaled TTS settings.

Motivation

  • Speech synthesis tests cross-modal transfer under the joint demands of faithful linguistic content and preservation of speaker identity from a reference utterance.Prior image-model adaptations largely remain within the visual domain, leaving transfer of image representations and generative priors to speech unresolved.Source: §1 · p. 1

Prior cross-modal evidence

  • Riffusion provides early evidence that image-acquired representations and generative priors can help model spectrograms for text-conditioned music generation.The paper distinguishes this music-generation precedent from its focus on speech synthesis.Source: §1 · p. 2

Research questions

  • Systematic vision-to-speech transfer is evaluated through three questions: whether image latents represent speech, whether image priors help over random initialization, and how competitive transfer becomes at scale.The evaluation separates representation transfer, initialization transfer, and downstream scaling.Source: §1 · p. 2

ImageTTS framework

  • ImageTTS converts log-mel spectrograms into pseudo-RGB inputs, encodes them with the frozen image VAE, and fine-tunes the pretrained FLUX.2 DiT for speech generation.The paper frames image representation transfer and generative-prior transfer as separable components.Source: §1 · p. 2

Evidence of transfer

  • The frozen image VAE preserves speech intelligibility and speaker identity, while image-pretrained DiT initialization improves convergence and linguistic accuracy over random initialization under matched training conditions.These results provide evidence for representation transfer and generative-prior transfer, respectively.Source: §1 · p. 2

Scaling to zero-shot and instruction-guided TTS

  • With approximately 100K hours of speech pretraining, ImageTTS-base supports zero-shot TTS, while ImageTTS-instruct enables natural-language control and outperforms comparable baselines.ImageTTS-instruct is further trained on CapSpeech.Source: §1 · p. 2

2 RELATED WORK

Related work covers flow- and diffusion-based zero-shot TTS, continuous speech representations, and natural-language instruction control. ImageTTS differs by adapting an image model’s latent space, multimodal architecture, and text encoder for speech tasks.

2.1 DIFFUSION AND FLOW MODELS FOR ZERO-SHOT TTS

  • ImageTTS avoids token-level alignment and combines modality-specific early representations with unified multimodal modeling in later blocks, unlike earlier shared-fusion approaches.Prior zero-shot systems include flow-matching mel infilling and Diffusion Transformers.Source: §2.1 · pp. 2–3

2.2 REPRESENTATION SPACES FOR SPEECH SYNTHESIS

  • Instead of training a speech-specific autoencoder, ImageTTS encodes pseudo-RGB mel-spectrograms with a frozen image VAE to test whether visual latents preserve speech information and support synthesis.Speech systems also model mel-spectrograms or use dedicated speech-specific continuous latent representations.Source: §2.2 · p. 3

2.3 INSTRUCTION-GUIDED TTS

  • ImageTTS reuses a pretrained image text encoder for transcripts and style descriptions to test whether image-generation pipelines can ground vocal descriptions in controllable speech.Instruction-guided TTS provides control over content and delivery, including style, emotion, and speaker attributes.Source: §2.3 · p. 3

3 METHOD

ImageTTS represents speech in a frozen image-VAE latent space and trains a conditional flow-matching DiT for zero-shot and instruction-guided synthesis. Its method combines task-specific conditioning with masked training and ODE-based inference.

3.1 SPEECH REPRESENTATION IN THE PRETRAINED IMAGE LATENT SPACE

  • Mel-spectrograms become pseudo-RGB inputs for the frozen FLUX.2 VAE: frequency and time map to image height and width, with resizing, padding, normalization, and channel replication.The VAE downsamples each spatial dimension by 8 before its latents are rearranged into 2 × 2 DiT patches; input dimensions must therefore be divisible by 16.Source: §3.1 · pp. 3–4

3.2 MULTI-TASK CONDITIONAL FLOW MATCHING

  • Conditional flow matching trains the DiT to predict a velocity field from noisy speech latents, transcript tokens, and an optional reference-speech latent.The noise level runs from Gaussian noise at σ = 1 to the clean target latent at σ = 0, and a task-specific binary mask selects loss positions.Source: §3.2 · p. 4
  • Inference starts from Gaussian noise and solves a conditional ODE across decreasing noise levels, with the number of function evaluations trading sampling accuracy against computational cost.The resulting latent is decoded as speech.Source: §3.2 · p. 4
  • Zero-shot TTS uses masked latent infilling: observed reference speech conditions the model while a selected region is generated, and inference retains only the continuation suffix.During inference, the reference-conditioning stream stays fixed while the target latent is updated over the requested duration.Source: §3.2 · pp. 4–5
  • Instruction-guided TTS generates speech from a transcript and natural-language style instruction, encoded together without reference-speech tokens.The complete target latent is used for flow-matching loss.Source: §3.2 · p. 5
  • Classifier-free guidance independently scales transcript conditioning for linguistic fidelity and reference-speech conditioning for speaker identity, while instruction-guided TTS applies guidance only to style.Training can drop transcript, reference-speech, or style conditions to support the corresponding guidance.Source: §3.2 · p. 5

4 ANALYZING TRANSFER FROM IMAGE GENERATION TO SPEECH

ImageTTS tests whether a frozen image VAE can represent speech and whether image-pretrained DiT weights help speech generation. Across reconstruction, zero-shot, and instruction-guided evaluations, results show representation and initialization benefits, with model performance varying by task and generation space.

4.1 SPEECH RECONSTRUCTION WITH A FROZEN IMAGE VAE

  • At F = 512, frozen-FLUX.2-VAE reconstruction reaches STOI 0.99 and SIM 0.95 on LibriTTS, while PESQ remains 0.57 below the vocoder-only baseline.The authors select F = 512 to balance reconstruction quality against latent sequence length; doubling resolution to 1024 gives smaller additional PESQ gains.Source: §4.1 · pp. 5–6, Table 1 · p. 6

4.2 EFFECT OF IMAGE-PRETRAINED INITIALIZATION ON SPEECH ADAPTATION

  • Image-pretrained DiT reaches at 100k updates approximately the training loss that random initialization reaches only after 150k updates.The models share the same architecture, frozen image-VAE latent space, and optimization schedule, isolating the initialization difference.Source: §4.2 · pp. 6–7, Figure 3 · p. 6
  • At 100k updates, image-pretrained initialization yields 2.7% WER versus 3.8% for random initialization, even after the random model trains 150k updates to comparable loss.The compared models share architecture, latent space, and optimization schedule; the image-pretrained model also maintains higher SIM across checkpoints.Source: §4.2 · p. 7, Figure 4 · p. 7

4.3 INTERACTION BETWEEN GENERATION SPACE AND PRETRAINED INITIALIZATION

  • At 100k updates, image-pretrained initialization reduces WER in both modeling spaces, while image-VAE latent modeling has higher DNSMOS and direct mel modeling higher speaker similarity across both initializations.The authors hypothesize that VAE latents provide a smoother, more structured prediction target, whereas direct mel modeling preserves more speaker-related spectral information.Source: §4.3 · pp. 7–8, Table 2 · p. 7

4.4 ZERO-SHOT TTS RESULTS

  • Among systems trained on comparable Emilia data, ImageTTS-base has the lowest text error in every evaluation setting and the highest MOS on both Seed-TTS sets: 4.08 English and 4.01 Chinese.Speaker similarity is lower than F5-TTS and MaskGCT on the Chinese Seed-TTS set; the authors connect this gap to speaker-identity information preserved by the frozen VAE.Source: §4.4 · p. 8, Table 3 · p. 8

4.4.1 EFFECT OF DUAL CLASSIFIER-FREE GUIDANCE

  • In the Seed-TTS test-en guidance sweep, (st, ss) = (2.5, 1.8) provides a favorable balance: WER reaches 0.55% at st = 2.5, while SIM peaks at 0.651 with ss = 1.8.Excessively strong text guidance can slightly reduce speaker similarity, and raising speech guidance beyond 1.8 provides little further benefit.Source: §4.4.1 · p. 8, Figure 5 · p. 9

4.5 INSTRUCTION-GUIDED TTS RESULTS

  • With 9K hours of CapSpeech instruction data, ImageTTS-instruct leads nearly all CapSpeech-trained metrics and reaches Style-ACC 0.66 and CLSP-Score 0.49 overall.The CapTTS baselines use 33.6K hours of instruction data; ImageTTS-instruct remains competitive with systems trained on other instruction datasets.Source: §4.5 · p. 9, Table 4 · p. 9

5 CONCLUSION

ImageTTS repurposes FLUX.2 for speech synthesis, with a frozen image VAE and pretrained DiT supporting representation and initialization transfer. At scale, it reports results on zero-shot and instruction-guided TTS, and identifies VAE adaptation as a future direction.

5 CONCLUSION

  • 0.99 STOI and 0.95 speaker similarity are achieved with the frozen FLUX.2 VAE in controlled LibriTTS experiments.Image-pretrained DiT initialization also improves adaptation efficiency and reduces WER relative to matched training from scratch.Source: §5 · p. 9
  • 1.12% WER on Seed-TTS test-en and Style-ACC 0.66 on CapSpeech are reported after scaling ImageTTS to 100K hours for zero-shot and instruction-guided TTS.The authors suggest these results indicate that pretrained image-generation models can transfer beyond vision.Source: §5 · pp. 9–10
  • Adapting the image VAE is identified as a future direction for improving speaker preservation and perceptual speech quality.The paper presents ImageTTS as modeling pseudo-RGB mel-spectrograms in the latent space of a frozen image VAE.Source: §5 · pp. 9–10

AI USE STATEMENT

This material describes the authors’ AI-use statement, the FLUX.2 pipeline and its speech-generation baselines. FLUX.2 combines text, image-latent, and Transformer components, while the baselines span several TTS architectures and training settings.

AI USE STATEMENT

  • The authors used generative AI for manuscript language editing and feedback, then independently checked the text and interpretations against results and literature.The authors state that they remain responsible for all claims and content.Source: AI USE STATEMENT · p. 10

A PRELIMINARIES: THE FLUX.2-KLEIN-4B PIPELINE

  • FLUX.2-klein-base-4B combines a Qwen3 text encoder, a VAE, and a 3.9B-parameter multimodal Transformer that predicts latent-sample velocity.The text encoder supplies contextual representations, and the VAE maps images between pixel and continuous latent spaces.Source: A PRELIMINARIES: THE FLUX.2-KLEIN-4B… · p. 13
  • The frozen Qwen3 encoder supplies concatenated hidden states from layers 9, 18, and 27, which a learned projection maps to the Transformer hidden dimension.The multiple layers provide contextual features at different abstraction levels.Source: A PRELIMINARIES: THE FLUX.2-KLEIN-4B… · p. 13
  • The pretrained VAE encodes images into continuous latent representations, which are patchified and flattened into tokens before generation and decoded back to pixel space afterward.The latent tokens are standardized and projected into Transformer hidden space; generation reverses the patchification.Source: A PRELIMINARIES: THE FLUX.2-KLEIN-4B… · p. 14
  • Its DiT has five double-stream blocks followed by twenty single-stream blocks, with joint attention enabling text and visual tokens to exchange information.The visual portion of the final sequence is projected back to latent dimensionality.Source: A PRELIMINARIES: THE FLUX.2-KLEIN-4B… · p. 14
  • Multi-axis rotary embeddings encode text and visual positions across four axes, while the flow timestep enters each block through adaptive normalization.Text uses a one-dimensional sequence axis and visual tokens use two-dimensional spatial coordinates.Source: A PRELIMINARIES: THE FLUX.2-KLEIN-4B… · p. 14

B BASELINES

  • F5-TTS uses flow matching with a DiT on about 95k Emilia hours, while MaskGCT generates semantic and acoustic tokens and shares ImageTTS-base’s 100k-hour bilingual Emilia corpus.The baselines also include Natural-language-guided PromptTTS (Leng et al., 2024) and instruction-driven VoxInstruct (Zhou et al., 2024).Source: B BASELINES · pp. 14–15
  • The compared systems also include Qwen3-TTS and Qwen3-TTS-VD (Hu et al., 2026), FireRedTTS3-Instruct (Shen et al., 2026), VoxCPM2 (Zhou et al., 2026), and CapTTS-AR/NAR (Wang et al., 2025a).Qwen3-TTS Base is used for zero-shot voice cloning and its VoiceDesign checkpoint for instruction-guided synthesis; CapTTS baselines use the same style-captioned data as ImageTTS-instruct.Source: B BASELINES · p. 15

C EVALUATION METRICS

The general TTS metrics cover perceptual quality, intelligibility, speaker identity, and recognition errors. They compare synthesized speech with references or estimate speech quality, depending on the measure.

C EVALUATION METRICS

  • PESQ compares synthesized speech perceptual quality with reference speech, while DNSMOS and UTMOS estimate or predict perceptual quality.PESQ is attributed to Rix et al. (2001), DNSMOS to Reddy et al. (2021), and UTMOS to Saeki et al. (2022).Source: §C.1 · p. 15
  • STOI evaluates intelligibility from temporal-structure comparisons, and Speaker Similarity measures identity preservation using cosine similarity between WavLM-TDNN embeddings.The cited works are Taal et al. (2010) for STOI and Chen et al. (2022) for Speaker Similarity.Source: §C.1 · p. 15
  • WER measures word-level recognition errors using a Conformer-Transducer, while CER measures character-level errors in Chinese speech using Paraformer-zh.The cited model references are Gulati et al. (2020) for WER and Gao et al. (2022) for CER.Source: §C.1 · p. 15

C.2 SUBJECTIVE METRIC FOR TTS

The evaluation passages specify a listener-rated MOS protocol, metrics for instruction-guided speech, and Riffusion-based spectrogram preprocessing. The instruction metrics cover style attributes and prompt–speech consistency, while preprocessing converts mel spectrograms to and from VAE-compatible RGB images.

C.2 SUBJECTIVE METRIC FOR TTS

  • MOS is averaged over 30 generated utterances—15 English and 15 Chinese—with 10 independent listeners rating each on a 1–5 scale.The reported average includes a 95% confidence interval.Source: §C.2 · p. 15

C.3 METRICS FOR INSTRUCTION-GUIDED TTS

  • Instruction-guided evaluation reports accuracy for age, gender, pitch, expressiveness, and aggregate Style-ACC, plus CLSP-Score for instruction–waveform semantic consistency.Style-ACC averages sample-level classification accuracy across evaluated style attributes; CLSP-Score averages cosine similarity between normalized text and audio embeddings.Source: §C.3 · p. 16
  • Duration accuracy is omitted because externally estimated durations make it partly reflect the estimation procedure rather than instruction-following ability.The reported style attributes are age, gender, pitch, and expressiveness.Source: §C.3 · p. 16

D RIFFUSION PREPROCESSING

  • Riffusion preprocessing pads a 100-bin Mel spectrogram to multiples of 32, transforms and converts it to RGB for VAE encoding, then reverses the steps and removes padding for BigVGAN.The transformations include normalization, a power transform, intensity inversion, and vertical flipping.Source: D RIFFUSION PREPROCESSING · p. 16

E IMAGE-PRETRAINED INITIALIZATION FOR SPEECH ADAPTATION

The initialization study compares image-pretrained and randomly initialized DiTs while holding the architecture, data, objective, and procedure constant. It evaluates fixed checkpoints and includes a longer random-initialization run that reaches comparable training loss.

E IMAGE-PRETRAINED INITIALIZATION FOR SPEECH ADAPTATION

  • Both models share the architecture, data, objective, and training procedure; only DiT initialization differs.The image-pretrained model uses FLUX.2 parameters, while the random model uses default initialization with a fixed seed.Source: E IMAGE-PRETRAINED INITIALIZATION FOR… · p. 16
  • At 150k updates, the randomly initialized model reaches comparable training loss with 3.8% WER, 0.76 speaker similarity, and 3.75 DNSMOS.The image-pretrained model is trained for 100k updates; the random model is trained for 150k.Source: E IMAGE-PRETRAINED INITIALIZATION FOR… · p. 16
  • Convergence is assessed at 30k, 40k, 50k, 80k, and 100k updates on the same 200 test utterances, with text and speech CFG scales fixed at 4 and 1.Training uses 16 NVIDIA A100 GPUs, BF16 mixed precision, AdamW, a 1×10^-4 learning rate, and effective batch size 128.Source: E IMAGE-PRETRAINED INITIALIZATION FOR… · p. 16

F EXPERIMENTAL SETUP FOR SCALING TASK

The scaling experiments use Emilia for zero-shot TTS and CapSpeech for instruction-guided TTS. ImageTTS-base is trained from FLUX.2, then further trained with a mixture of instruction-guided and zero-shot examples to produce ImageTTS-instruct.

F EXPERIMENTAL SETUP FOR SCALING TASK

  • Zero-shot training uses approximately 100k hours of English and Chinese Emilia speech, evaluated on 1,088 English and 2,020 Chinese Seed-TTS samples.Instruction-guided training uses 3.47 million CapSpeech pairs, totaling approximately 9K hours from MLS-English, Emilia-English, and Common Voice; its collection matches CapTTS's but differs in size.Source: F EXPERIMENTAL SETUP FOR SCALING TASK · p. 17
  • ImageTTS-base is initialized from FLUX.2 and trained for 180k updates; 50k further updates with a 1:1 sample-count mixture yield ImageTTS-instruct, which supports both tasks.Training uses 64 NVIDIA A100 GPUs, BF16 mixed precision, AdamW, a 2×10^-4 learning rate, and effective batch size 256.Source: F EXPERIMENTAL SETUP FOR SCALING TASK · p. 17
  • Inference conditions on a reference latent in the target latent's temporal prefix and iteratively samples with an Euler ODE solver, dual classifier-free guidance, and NFE = 50.All latent positions, including the reference prefix, are updated; the decoded mel-spectrogram is converted to waveform with BigVGAN-v2.Source: F EXPERIMENTAL SETUP FOR SCALING TASK · p. 17

G INTERNAL ENGLISH TEST SET

The internal English test set uses screened reference utterances and GPT-5-generated prompts. The duration-estimation procedure sets speaking rates from reference speech for zero-shot TTS and from style instructions for instruction-guided TTS.

G INTERNAL ENGLISH TEST SET

  • 20 reference utterances from 11 speakers are screened for clear, natural pronunciation, and each receives five GPT-5 prompts spanning sentence structures and linguistic complexity.The set averages 4.15 UTMOS and is slated for release after anonymous review.Source: G INTERNAL ENGLISH TEST SET · p. 17

H DURATION ESTIMATION

  • For zero-shot TTS, speaking rate is estimated from valid characters or phonemes per second, then constrained to 3–6 Chinese characters/s for Chinese references.Leading and trailing silence is removed before estimating the rate.Source: H DURATION ESTIMATION · p. 18
  • For instruction-guided TTS, duration is set from the described speaking rate using base rates of 5 Chinese characters/s or 14 English phonemes/s and specified style multipliers.Because this procedure explicitly determines duration rather than the model predicting it, duration control is excluded from the style-control accuracy metric.Source: H DURATION ESTIMATION · p. 18

I AN ANALYSIS OF TEXT ENCODER

In four selected Chinese cases, punctuation edits coincide with lower character and ASR-derived syllable error rates, while the analysis cautions that these checks do not directly verify audio phonetics or isolate tokenization effects. An exploratory English analysis reports an association between multi-token words and higher ASR error rates.

I AN ANALYSIS OF TEXT ENCODER

  • The tone-aware PSER converts reference and ASR text into tone-numbered syllables and computes normalized Levenshtein distance over complete sentences.Aggregate scores pool edit counts and reference lengths.Source: I AN ANALYSIS OF TEXT ENCODER · p. 18
  • CER decreases from 21.21% to 1.52%, and ASR-derived PSER decreases from 13.64% to 0.00% across four selected Chinese cases.The remaining “赖”/“癞” substitution maps to the same Pinyin syllable, lai4, so it adds no PSER error; this indicates converted-Pinyin agreement, not direct phonetic verification.Source: I AN ANALYSIS OF TEXT ENCODER · p. 18
  • Punctuation changes how the same Chinese character sequence is represented, but these diagnostic edits do not isolate tokenization as the cause because commas also affect context and phrasing.The examples do not establish improved prosodic naturalness or generalization beyond the selected cases.Source: I AN ANALYSIS OF TEXT ENCODER · p. 18
  • The exploratory English analysis finds higher ASR word-error rates for words represented by multiple BPE tokens, but does not separate tokenization from word length, frequency, or proper-name status.Source: I AN ANALYSIS OF TEXT ENCODER · p. 19

Related papers

Loading 2610.34807v1…