PaperlayerView exact revision on arXiv

Source-linked AI summary

Local Prototype Reconstruction for Text-Compatible Speech-to-LLM Bridge Pretraining

Xinnian Zhao, Chia-Hua Wu, Pu Wang, Hugo Van Hamme

arXiv:2610.11159v1cs.CLcs.SD

TL;DR

Speech-to-LLM bridges need reusable pretraining, but global alignment alone does not determine whether individual bridge embeddings fit the frozen LLM's lexical neighbourhoods. The paper introduces LPR, a training-only token-level regularizer, and reports improved transfer across multilingual ASR and speech translation, with its diagnostic correlating with downstream gains.

  • Problem

    Global speech-text alignment does not establish whether individual bridge embeddings are locally compatible with the frozen LLM lexical space, a property relevant to reusable speech interfaces.

    Source: §I · p. 1

  • Method

    LPR adds a training-only token-level regularizer that reconstructs aligned bridge embeddings from local neighbourhoods of frozen LLM token embeddings.

    Source: §I · p. 2, B. Local Prototype Reconstruction · p. 3

  • Results

    LPR improves transfer across multilingual ASR and speech translation, especially under low-resource adaptation, and the independent compatibility diagnostic correlates with downstream gains.

    Source: §I · p. 2, §IX · p. 6

  • Takeaways & Limitations

    Local lexical manifold compatibility is a useful property to measure and encourage when pretraining reusable SpeechLLM bridges.

    Source: §IX · p. 6

Abstract

from arXiv · show

Speech-to-LLM systems often connect a frozen speech encoder to a frozen large language model (LLM) through a small trainable bridge. The bridge is usually treated as plumbing, but it in fact defines the geometry of the speech-to-LLM interface, and the pretraining objective decides whether that interface provides a reusable initialization for downstream tasks. We study a transferable bridge through two complementary properties: global alignment with the text side, and local lexical manifold compatibility, where bridge embeddings remain close to the frozen LLM's input-embedding neighbourhoods. We make this property measurable with a fixed, head-free, timestamp-free diagnostic that applies to any objective, and show that next-word prediction (NWP) and sentence-level contrastive pretraining do not fully capture token-level lexical compatibility. We then introduce Local Prototype Reconstruction (LPR), a lightweight training-only regularizer that requires each aligned bridge token to be reconstructable from a small neighbourhood of frozen LLM token embeddings, with a hard single-prototype anchor as its limiting case. On multilingual ASR and speech translation, LPR improves transfer, with the largest gains on translation and low-resource adaptation. Crucially, our independent diagnostic correlates with downstream gains across objectives, suggesting that lexical manifold compatibility is predictive of reusability for speech-to-LLM bridges.

I. INTRODUCTION

With a frozen speech encoder and LLM, the bridge shapes the interface, raising the question of which pretraining objective yields a reusable representation. The paper distinguishes global alignment from local lexical compatibility and introduces LPR to encourage both.

I. INTRODUCTION

  • The paper treats global semantic alignment and local lexical manifold compatibility as complementary properties of a reusable speech-to-LLM interface.The motivation is that the LLM consumes token-like sequences, not only pooled utterance vectors.Source: §I · p. 1
  • NWP trains generation and contrastive pretraining aligns pooled utterances, but neither fully specifies token-level compatibility with the frozen LLM lexical space.Contrastive retrieval can be near-perfect while token-level compatibility remains low.Source: §I · p. 1, Figure 1 · p. 1
  • LPR is a continuous, training-only regularizer that reconstructs transcript-aligned bridge tokens from local frozen-LLM embedding neighbourhoods.A hard single-prototype anchor is its limiting case.Source: §I · pp. 1–2
  • A fixed, head-free, timestamp-free diagnostic measures interface compatibility uniformly across NWP, contrastive, and LPR checkpoints.It separates utterance-level retrieval from token-level cosine to nearby LLM embeddings.Source: Figure 1 · p. 1, §I · p. 2
  • LPR improves transfer on multilingual ASR and speech translation, with the largest gains under low-resource adaptation and on translation.The independent compatibility diagnostic also correlates with downstream gains across objectives.Source: §I · p. 2

II. RELATED WORK

Prior SpeechLLM work often evaluates connectors through downstream generation, while the geometry of the speech-to-LLM interface receives less direct attention. This paper positions LPR and its diagnostic alongside work on contrastive alignment, tokenization, and representation geometry.

Speech-to-LLM bridges and pretraining objectives

  • SpeechLLM connectors commonly use autoregressive or instruction-following objectives –, –[22], but are typically evaluated by generation quality rather than interface geometry.Recent connector comparisons include –,,.Source: §II · p. 2
  • Sentence-level contrastive pretraining aligns pooled speech and text representations, while leaving the compatibility of individual bridge embeddings with the LLM lexical space as a complementary question.LPR uses this contrastive objective as its global-alignment backbone.Source: §II · p. 2

Cross-modal alignment, speech tokenization, and lexical interfaces

  • Speech tokenization methods make the interface more token-like –[27], but discretization can discard continuous information and tie the bridge to a tokenizer.Optimal-transport translation and knowledge distillation [29] instead target hidden-state or output-distribution alignment.Source: §II · p. 2

Representation geometry and interface diagnostics

  • The diagnostic adapts geometric probing of language representations to test whether speech-derived tokens occupy neighbourhoods used by the frozen LLM for lexical input.It is model-agnostic, head-free, and timestamp-free, and can compare different pretraining objectives under the same probe.Source: §II · p. 2

III. METHOD

The method connects a frozen speech encoder to a frozen LLM through a trainable bridge and combines global sentence-level alignment with token-level compatibility. Its contrastive backbone aligns pooled speech and transcript representations, while LPR adds token-level reconstruction from local LLM embedding neighbourhoods.

III. METHOD

  • The architecture maps frozen speech-encoder features through a trainable bridge into continuous embeddings for a frozen LLM.Only the bridge and small LPR alignment projections are updated.Source: §III · p. 2

A. Sentence-Level Contrastive Backbone

B. Local Prototype Reconstruction

Local Prototype Reconstruction (LPR) adds token-level lexical constraints while keeping the bridge continuous. It aligns bridge outputs to transcript positions and reconstructs each aligned token from a local neighbourhood of frozen LLM embeddings.

B. Local Prototype Reconstruction

  • LPR aligns bridge outputs to transcript positions because bridge and transcript lengths need not match.The alignment projections compute weights, while the aligned representation remains a weighted sum of original bridge outputs evaluated in the LLM input-embedding space.Source: B. Local Prototype Reconstruction · p. 3
  • Each transcript token’s prototype set includes its gold LLM embedding and nearest neighbours from a fixed, offline candidate pool.Neighbours are ranked by cosine similarity between normalized embeddings, focusing reconstruction on the gold token’s local lexical neighbourhood.Source: B. Local Prototype Reconstruction · p. 3
  • The reconstruction is a local convex combination of frozen LLM token embeddings, encouraging lexical-interface compatibility without forcing a discrete token match.The aligned representation is compared with the gold embedding and its prototypes in the LLM input-embedding space.Source: B. Local Prototype Reconstruction · p. 3
  • The anchor term grounds aligned tokens directly to the gold embedding, while the reconstruction term tests compatibility with its local neighbourhood.When K = 1, the reconstruction loss reduces to the anchor loss, making hard anchoring and soft reconstruction limiting cases on the same lexical-grounding axis.Source: B. Local Prototype Reconstruction · pp. 3–4

C. Training Objective and Inference

The objective combines pretraining regularizers with the contrastive baseline. Alignment, prototype retrieval, and reconstruction are training-only computations; downstream models use the trained bridge with the frozen speech encoder and LLM.

C. Training Objective and Inference

  • Transcript-conditioned alignment, prototype retrieval, and reconstruction are used only to compute pretraining regularizers.The objective heading identifies the full pretraining objective, but the supplied passage does not specify its equation terms.Source: C. Training Objective and Inference · p. 4
  • Setting both LPR weights to zero recovers the contrastive baseline with the same architecture and training pipeline.This provides the stated baseline comparison for the LPR objective.Source: C. Training Objective and Inference · p. 4
  • Downstream fine-tuning and inference use only the frozen speech encoder, trained bridge, and frozen LLM.The alignment, retrieval, and reconstruction computations do not form part of downstream model use.Source: C. Training Objective and Inference · p. 4

IV. MEASURING INTERFACE COMPATIBILITY

The paper evaluates interface compatibility with a fixed probe applied uniformly across NWP, contrastive, and LPR checkpoints. It pairs global retrieval with token-level reconstruction, rank, and entropy diagnostics, interpreting these alongside downstream transfer.

IV. Measuring Interface Compatibility

  • The fixed, head-free probe uses transcript-conditioned parameter-free soft alignment and requires transcripts but no word timestamps.It is applied identically to checkpoints trained with NWP, contrastive pretraining, and LPR.Source: §IV · p. 4
  • Utterance Retrieval R@1 measures global alignment by retrieving paired speech and text representations among in-batch candidates.Source: §IV · p. 4
  • Manifold Reconstruction Error measures the squared distance between normalized aligned speech tokens and their prototype reconstructions, with lower error indicating better local reconstruction.Source: §IV · p. 4
  • Gold-token Rank evaluates whether aligned speech tokens identify their gold embedding, while prototype entropy measures how compactly reconstruction uses local prototypes.Source: §IV · p. 4
  • Alignment entropy measures whether transcript tokens associate with localized or broad bridge-output spans; diagnostics are interpreted with downstream transfer rather than alone.The central question is whether a fixed diagnostic applied uniformly across objectives predicts ASR and ST gains.Source: §IV · p. 4

V. EXPERIMENTAL SETUP

Experiments use a shared multilingual bridge initialization for ASR and speech translation on five CoVoST2 source languages. The setup compares pretraining objectives under matched data, architecture, and compute conditions.

V. Experimental Setup

  • CoVoST2 experiments use German, French, Spanish, Italian, and Portuguese speech with English targets, totaling about 431 training, 83 development, and 88 test hours.A 10% low-resource fine-tuning subset contains approximately 43 hours.Source: §V · p. 4
  • The system freezes Whisper-large-v3 as the speech encoder and Llama-3-8B-Instruct as the LLM, with a trainable 177M-parameter window-level Q-Former bridge.The comparison includes no pretraining, NWP, sentence-level contrastive pretraining, and LPR variants built on the contrastive backbone.Source: §V · p. 4
  • Each objective pretrains one multilingual bridge on five languages, then separately fine-tunes ASR and ST from that same initialization.Downstream fine-tuning is multilingual, with task and language specified by prompts; the no-pretraining baseline skips bridge pretraining.Source: §V · p. 4
  • All objectives share data splits, bridge architecture, precision, and compute budget; pretraining runs up to six epochs on one H100 GPU.The stated setup uses batch size 32, while LPR uses K = 16 prototypes and the contrastive baseline sets all LPR weights to zero.Source: §V · p. 4

VI. RESULTS

LPR provides the strongest downstream results across ASR and speech translation, with the largest gains on translation and under low-resource adaptation. The evaluations compare separately fine-tuned tasks initialized from the same pretrained multilingual bridge.

VI. RESULTS

  • Downstream ASR and ST are fine-tuned separately from the same pretrained multilingual bridge initialization; Table II’s fixed probe distinguishes utterance retrieval from token-level compatibility.The downstream comparison therefore assesses reuse of the bridge initialization across tasks; the diagnostics are computed on the multilingual development set.Source: §VI · p. 4, Table II · p. 5
  • 6.8% relative WER reduction lowers average full-resource ASR WER from 7.75% without pretraining to 7.22% with LPR (full).Under low-resource adaptation, LPR reduces average WER from 10.28% to 9.08%.Source: §VI · p. 5
  • 41.11 BLEU with LPR improves on 39.09 BLEU without pretraining in full-resource ST; under low-resource adaptation, LPR reaches 39.49 BLEU versus 36.55 BLEU with contrastive pretraining.NWP falls below the no-pretraining baseline in low-resource ST.Source: §VI · p. 5
  • LPR’s margin over contrastive pretraining increases in the 10% low-resource setting, especially for ST, where fine-tuning has less capacity to repair a weak interface.The authors caution that these results do not establish local lexical compatibility as the only factor.Source: §VI · p. 5
  • LPR gives the largest average relative improvement in ASR and ST across five source languages, with a larger margin in low-resource adaptation.The table caption reports LPR best in all columns, with gains largest on low-resource ST.Source: Figure 3 · p. 5, Table I · p. 5

VII. ANALYSIS: WHY IT TRANSFERS

The analysis separates global speech-text alignment from token-level lexical compatibility and tests whether a fixed probe tracks downstream transfer. Lower MRE is associated with greater gains, especially for ST and low-resource adaptation.

VII. ANALYSIS: WHY IT TRANSFERS

  • Contrastive pretraining and all LPR variants reach nearly perfect utterance retrieval, but this pooled measure cannot distinguish them despite differences in downstream ST and low-resource behavior.NWP remains lower on utterance retrieval, which measures pooled speech-text retrieval rather than conditional generation quality.Source: §VII · p. 6
  • Lower MRE is associated with larger downstream gains: full-resource correlations are ASR r=−0.51 and ST r=−0.69, versus low-resource ASR r=−0.90 and ST r=−0.88.The low-resource setting is especially informative because fine-tuning has less capacity to repair a weak interface.Source: §VII · p. 6
  • The MRE trend includes NWP and contrastive checkpoints that do not optimize MRE, so the fixed diagnostic ranks independently trained objectives in a way that tracks downstream transfer.The stronger correlations for ST and low-resource adaptation match the reported pattern that local lexical compatibility matters more for transfer beyond transcription or with limited data.Source: §VII · p. 6

VIII. DISCUSSION

The fixed diagnostic separates utterance-level alignment from token-level compatibility, and its compatibility measure tracks downstream gains most strongly in low-resource adaptation. Hard lexical anchoring and soft neighbourhood reconstruction provide complementary constraints, while LPR leaves inference unchanged.

VIII. DISCUSSION

  • The fixed probe distinguishes utterance-level semantic alignment from token-level lexical compatibility, which can explain different low-resource and translation behavior despite similar global retrieval.Contrastive pretraining captures global alignment well, while LPR mainly improves token-level compatibility.Source: §VIII · p. 6
  • Lower MRE is associated with larger downstream gains, most strongly in low-resource adaptation, where the pretrained bridge has less opportunity to be reshaped.The discussion says this setting makes compatibility with the frozen LLM more consequential.Source: Figure 4 · p. 6, §VIII · p. 6
  • Hard anchoring maximizes gold-token retrieval, while soft reconstruction targets a local lexical neighbourhood; their combination gives the best downstream results in these experiments.The two variants provide complementary constraints: direct lexical grounding and neighbourhood-level reconstruction.Source: §VIII · p. 6
  • LPR uses transcripts during supervised pretraining, but deployment uses only the frozen speech encoder, trained bridge, and frozen LLM, leaving inference unchanged from the contrastive baseline.Source: §VIII · p. 6

IX. CONCLUSION

The conclusion frames bridge geometry as relevant to transfer: global alignment alone does not establish local lexical compatibility. LPR adds a training-only token-level constraint, and its gains across multilingual ASR and speech translation are especially notable in low-resource adaptation.

IX. CONCLUSION

  • Across multilingual ASR and speech translation, LPR improves transfer, especially under low-resource adaptation.Source: §IX · p. 6
  • Sentence-level contrastive pretraining provides strong global alignment, but does not fully determine local compatibility with the frozen LLM lexical space.Source: §IX · p. 6
  • LPR adds token-level compatibility through a continuous, training-only prototype reconstruction regularizer.Source: §IX · p. 6
  • The independent compatibility diagnostic correlates with downstream gains, supporting local lexical compatibility as a useful property to measure and encourage in reusable SpeechLLM bridges.Source: §IX · p. 6

Related papers

Loading 2610.11159v1…