PaperlayerView exact revision on arXiv

Source-linked AI summary

SPEAR-Gen: Generation-Aware Pre-training for Unified Speech Representations

Xiaoyu Yang, Arthur Hinsvark, Antonios Alexos, Osama Hanna, Philip C. Woodland, Yiting Lu

arXiv:2609.34147v1eess.AScs.SD

TL;DR

Speech understanding and generation require representations that balance linguistic content with paralinguistic and acoustic cues, but existing objectives and models often specialize or retain a trade-off. SPEAR-Gen combines task-aligned feature aggregation with coarse-to-fine generative modeling in one representation. Experiments report strong understanding and resynthesis balance, with generation-aware training improving resynthesis while preserving linguistic performance.

  • Problem

    Speech representations and training objectives often specialize in understanding or generation, despite speech carrying linguistic, paralinguistic, and acoustic information relevant to both.

    Source: §1 · p. 1

  • Method

    SPEAR-Gen fuses task-aligned features across a frozen encoder into masked-prediction targets and combines log-Mel reconstruction with residual flow matching.

    Source: §1 · p. 1, §3.1 · p. 2, §3.2.2 · p. 2

  • Results

    Experiments show a strong understanding–generation balance in one representation, with generation-aware training improving resynthesis while preserving linguistic performance and strengthening paralinguistic information.

    Source: §1 · p. 1, §6 · p. 4

  • Takeaways & Limitations

    The experiments support using a single representation for speech understanding and generation within the evaluated SUPERB and resynthesis settings.

    Source: §1 · p. 1, §6 · p. 4

Abstract

from arXiv · show

Speech understanding and generation place different demands on speech representations, and existing models are typically optimised towards one capability or the other. To reduce this gap, we introduce SPEAR-Gen, a speech representation model that learns a single representation for both capabilities. Task-aligned feature aggregation consolidates complementary linguistic and paralinguistic information across a frozen encoder into discrete targets for masked prediction, while a coarse-to-fine objective combines log-Mel reconstruction with residual flow matching to preserve spectral structure and fine-grained acoustic variation. Experiments on SUPERB and speech resynthesis show that SPEAR-Gen maintains strong understanding performance while substantially improving resynthesis quality and speaker preservation. These results demonstrate that a single speech representation can effectively support both understanding and generation.

1. INTRODUCTION

Speech representations are often optimized for understanding or generation, although speech carries linguistic, paralinguistic, and acoustic information needed across both. SPEAR-Gen combines task-aligned targets and coarse-to-fine generation objectives in one representation, achieving a strong balance across understanding and resynthesis.

Motivation

  • Speech carries linguistic content alongside paralinguistic and acoustic cues, motivating a shared representation for diverse applications and future speech language models.Understanding-focused learning often prioritizes linguistic content, whereas generation also requires faithful preservation of acoustic cues.Source: §1 · p. 1
  • Understanding-oriented objectives can bias representations toward linguistic content, while reconstruction objectives preserve acoustic information but show weak understanding performance.Prior approaches either use separate representations or retain a trade-off between the capabilities.Source: §1 · p. 1

Approach

  • SPEAR-Gen uses task-aligned feature aggregation to fuse linguistic and paralinguistic cues across a frozen encoder into masked-prediction targets, alongside mel reconstruction and residual flow matching.Mel reconstruction retains dominant spectro-temporal structure, while residual flow matching supervises nuanced acoustic variation.Source: §1 · p. 1

Results

  • Experiments on SUPERB and speech resynthesis show a strong balance in one representation; generation-aware training improves resynthesis while preserving linguistic performance and strengthening paralinguistic information.Compared with WavCube, SPEAR-Gen achieves competitive resynthesis quality and stronger understanding performance.Source: §1 · p. 1

2. RELATED WORK

Prior work explores masked prediction and joint approaches to speech understanding and generation, but differs in how it constructs targets, combines tasks, and trains representations.

SPEAR and SPEAR-Gen

  • SPEAR-Gen extends SPEAR by constructing MVQ targets from fused representations across layers and jointly optimizing predictive and generative objectives.SPEAR learns understanding-oriented representations through masked prediction of MVQ tokens.Source: §2 · p. 1

Joint understanding and generation

  • OLIVE [12] combines latent prediction with waveform reconstruction but uses different feature levels for understanding and synthesis; UniWav uses flow-matching generation and aggregates encoder layers.These approaches provide related examples of joint learning with different representation strategies.Source: §2 · pp. 1–2

WavCube

  • WavCube [14] adapts an understanding-pretrained encoder into a semantic–acoustic model, whereas SPEAR-Gen trains its representation encoder from scratch with predictive and generative objectives jointly optimized.Figure 1 depicts SPEAR-Gen’s task-aligned feature fusion and combined masked-prediction and generative training.Source: §2 · p. 2, Figure 1 · p. 2

3. SPEAR-GEN

SPEAR-Gen trains one encoder representation for masked prediction and acoustic generation. It builds task-aligned discrete targets from layer-wise features, then combines log-Mel reconstruction and residual flow matching with the predictive objective.

3.1. Task-Aligned Feature Aggregation for MTP

  • TAFA normalizes and concatenates layer-wise features, projects them into a shared space, and learns a task-shared fusion using speech–text supervision while the SSL encoder remains frozen.The fused representation is quantised into MVQ targets for masked prediction, shaping the encoder output directly without downstream layer aggregation.Source: §3.1 · p. 2

3.2. Two-Stage Coarse-to-Fine Generative Modelling

  • The generative model combines an ℓ1 log-Mel objective for dominant spectro-temporal structure with flow matching over the residual for finer acoustic variation.This adds acoustic-detail supervision not explicitly required by masked prediction.Source: §3.2 · p. 2

3.2.1. Log-Mel Reconstruction

  • The mel decoder predicts a normalized log-Mel spectrogram from encoder output H, and an ℓ1 loss against the target encourages H to retain dominant spectro-temporal structure.The reconstruction target is the mean–variance normalized log-mel spectrogram.Source: §3.2.1 · p. 2

3.2.2. Residual Flow Matching

  • Residual flow matching models the difference between the target mel and the stop-gradient mel prediction, conditioning on H to capture cues absent from the coarse prediction.The decoder predicts target velocity from a noise-to-residual interpolation; noisier inputs are favored to encourage fine-grained acoustic detail.Source: §3.2.2 · p. 2

3.3. Unifying Predictive and Generative Objectives

  • Pre-training jointly optimizes masked prediction and generative losses, with λ weighting the generative objective and γ weighting residual flow matching.Masking is bypassed with probability pclean to reduce mismatch between masked pre-training and clean-input inference.Source: §3.3 · p. 3

4. EXPERIMENTAL SETUP

SPEAR-Gen uses a shared encoder and generative decoders, with TAFA alignment data kept separate from unlabelled pre-training data. Evaluation tests the final-layer representation for both understanding and resynthesis.

4.1. Model and Pre-training Configuration

  • Zipformer encodes 128-d log-Mel features at 100 Hz into 50 Hz representations; Base and Large use 95M and 327M encoder parameters, respectively.Both mel and flow-matching decoders use the same lightweight ConvNeXt architecture, with approximately 15M parameters each.Source: §4.1 · p. 3

4.2. Training Data

  • About 350 hours of labelled speech support TAFA alignment, while unannotated LibriSpeech and LibriHeavy train Base and Large, respectively.The alignment corpora are disjoint from SUPERB evaluation datasets, so evaluation transcripts and speaker identities are not exposed through TAFA alignment.Source: §4.2 · p. 3, Table 1 · p. 3

4.3. Evaluation Protocols

  • Evaluation probes the final-layer representation on SUPERB understanding tasks and tests resynthesis with a HiFi-GAN vocoder conditioned on that representation.SUPERB tasks span linguistic and paralinguistic information; resynthesis uses perceptual-quality, intelligibility, and speaker-preservation metrics on LibriSpeech test-clean.Source: §4.3 · p. 3

5. EXPERIMENTAL RESULTS

SPEAR-Gen balances final-layer understanding and resynthesis, outperforming understanding-oriented counterparts on acoustic recoverability while retaining strong understanding. Ablations distinguish the contributions of TAFA targets, MTP, and coarse-to-fine generative modelling.

5.1. Understanding and Resynthesis Performance

  • SPEAR-Gen Base scores stronger across reported SUPERB tasks than WavCube with comparable resynthesis quality and fewer encoder parameters; Large matches WavCube in PESQ and STOI while improving ViSQOL, WER, SIM, and UTMOS.The combined results indicate a better balance between understanding and acoustic recoverability within one representation.Source: §5.1 · pp. 3–4
  • PESQ rises from 1.20 to 2.94, ViSQOL from 1.70 to 4.12, and SIM from 0.467 to 0.892, while WER falls from 2.61 to 2.22 versus SPEAR.On SUPERB, Base improves SPEAR on ASR, KWS, ER, SID, and SV, with modest changes on PR, IC, and SD.Source: §5.1 · p. 3, Table 3 · p. 4, Table 2 · p. 4

5.2. Ablation Studies

  • TAFA improves all reported metrics over single-layer targets and outperforms uniform layer averaging; adding generative supervision with TAFA raises ViSQOL from 1.86 to 4.12 and SIM from 0.555 to 0.892.Generative-only training yields the highest ViSQOL and SIM but substantially weaker SUPERB performance, supporting MTP's role in preserving understanding.Source: §5.2 · p. 4, Table 4 · p. 4
  • Doubling the mel decoder from 15M to 30M parameters gives mixed results, while applying flow matching directly to the full mel target is less effective for resynthesis.The comparison includes alternative generative objectives and decoder parameter counts.Source: §5.2 · p. 4, Table 5 · p. 4
  • The proposed coarse-to-fine objective gives the strongest understanding–resynthesis balance, outperforming a capacity-matched mel decoder with only ℓ1 loss.The results support residual flow matching beyond simply increasing decoder capacity.Source: §5.2 · p. 4, Table 5 · p. 4

6. CONCLUSIONS

SPEAR-Gen combines linguistic, paralinguistic, and acoustic information in one representation for speech understanding and generation. Experiments show a balance between understanding and resynthesis, with generative supervision improving several understanding tasks in a controlled ablation.

6. CONCLUSIONS

  • Task-aligned feature aggregation fuses linguistic and paralinguistic cues into prediction targets, while coarse-to-fine generative modelling encourages the same representation to preserve acoustic detail.Source: §6 · p. 4
  • SPEAR-Gen balances speech understanding and speech resynthesis within a single representation, and generative supervision improves ASR, ER, and SV in the controlled ablation.Source: §6 · p. 4
  • Future work will investigate SPEAR-Gen’s suitability as a prediction target for text-conditioned speech generation.Source: §6 · p. 4

Related papers

Loading 2609.34147v1…