PaperlayerView exact revision on arXiv

Source-linked AI summary

INSPIRE: A Benchmark for Instruction-Aware Speech Retrieval

Chen-An Li, Hung-yi Lee

arXiv:2608.16203v1cs.SDcs.CLeess.AS

TL;DR

Existing speech retrieval systems use rigid similarity matching despite instructions that can target semantic, speaker, stylistic, or environmental attributes. INSPIRE benchmarks instruction-aware retrieval across these intents and finds that no current paradigm robustly handles them all, with semantic and acoustic strengths divided across approaches.

  • Problem

    Existing speech retrieval systems use fixed similarity functions even though users may define relevance through heterogeneous semantic, speaker, stylistic, and environmental attributes.

  • Method

    INSPIRE is a benchmark that evaluates diverse speech-retrieval model families under natural-language instructions spanning semantic, speaker, style, background, and multi-attribute intents.

  • Results

    No evaluated method robustly handles all retrieval intents: cascaded approaches favor semantic matching, speech-based models preserve acoustic properties, and complex multi-attribute scenarios remain difficult.

  • Takeaways & Limitations

    The findings motivate unified multimodal architectures and instruction-aware training that jointly preserve fine-grained acoustic information and follow compositional instructions.

  • Takeaways & Limitations

    Caption-then-embed pipelines can lose fine-grained acoustic cues by compressing rich speech into text.

Abstract

from arXiv · show

Existing speech retrieval systems rely on fixed similarity matching and cannot adapt to diverse user intents. We introduce INSPIRE, the first benchmark for instruction-aware speech retrieval, in which natural-language instructions dynamically specify relevance criteria, including semantic content, speaker identity, speaking style, environmental sounds, and their combinations. We evaluate four retrieval paradigms: large audio-language models, cascaded pipelines, self-supervised speech models, and contrastive audio-language models. Our results reveal that no current method robustly handles all retrieval intents. Text-based approaches perform relatively better at semantic retrieval but struggle with paralinguistic attributes, while speech-based models are moderately better at capturing acoustic properties but falter at following instructions. These findings highlight the need for unified architectures capable of instruction-aware speech retrieval.

1. Introduction

INSPIRE formalizes instruction-aware speech retrieval as a benchmarked problem in which natural-language instructions determine which spoken documents are relevant across heterogeneous attributes. The introduction motivates this task by showing that systems must preserve fine-grained acoustic detail while adapting to linguistic, paralinguistic, and environmental cues.

  • Motivation: Traditional speech retrieval matches documents using fixed acoustic or semantic similarity, whereas real-world users often specify relevance through instructions.Examples include finding calls with the same frustrated tone or other instruction-defined properties.
  • Motivation: Instruction-aware speech retrieval must interpret linguistic content, paralinguistic characteristics, and environmental context while weighting these cues according to the instruction.Because queries are speech signals and relevant attributes are heterogeneous, representations must retain fine-grained acoustic detail without sacrificing content information.
  • Retrieval intents: Different instructions paired with the same spoken query can target content, speaker, style, background, or combinations of these attributes.The benchmark therefore covers both single-attribute and composite retrieval intents.
  • Contributions: INSPIRE is introduced as the first benchmark for instruction-aware speech retrieval, spanning semantic, speaker, style, background, and multi-attribute intents.It evaluates whether systems follow natural-language instructions when selecting relevant spoken documents.
  • Contributions: The benchmark provides a unified evaluation protocol across large audio-language models, cascaded ASR-and-captioning pipelines, self-supervised speech representations, and contrastive audio-language architectures.The authors frame instruction-aware speech retrieval as a research problem requiring architectures and training paradigms specifically tailored to it.

2. Related Work

Prior speech retrieval and representation benchmarks largely use fixed similarity objectives or downstream tasks, while instruction-aware retrieval has been established mainly in text and image domains. INSPIRE brings natural-language, instruction-conditioned relevance to speech retrieval, including semantic, paralinguistic, and compositional constraints.

  • Speech retrieval: Early speech retrieval progressed from query-by-example spoken term detection to speech-to-speech and text-to-speech retrieval over large collections.These methods typically define relevance through acoustic or semantic similarity with a fixed retrieval objective per spoken query.
  • Speech retrieval: Spoken question answering identifies answer spans for questions but does not address retrieval based on paralinguistic attributes or multi-attribute compositions.INSPIRE includes SQA-like semantic intents while adding speaker-, style-, and environment-based constraints.
  • Instruction-aware retrieval: Instruction-aware retrieval is established in text and image retrieval, but instruction-following has not been systematically studied for speech retrieval.INSPIRE applies natural-language matching instructions to speech, where relevance can depend on linguistic and paralinguistic cues.
  • Representation benchmarks: Speech and audio representation benchmarks evaluate fixed downstream tasks and probing suites but generally do not test instruction-conditioned relevance.INSPIRE complements NOSS, SUPERB, HARES, MSEB, and MAEB by directly evaluating retrieval behavior under natural-language instructions.

3. Design of INSPIRE

INSPIRE defines retrieval as ranking spoken documents conditioned on both a spoken query and a natural-language instruction specifying relevance constraints. Its four independent subsets progressively evaluate conversational continuity, speaker identity, speaking style, and compositional multi-attribute matching.

  • Retrieval formulation: Each search pairs a spoken query with an instruction that may constrain semantic content, speaker identity, vocal style, or acoustic environment.The system ranks documents using an instruction-conditioned score, with relevant documents required to outrank all irrelevant documents.
  • Benchmark subsets: INSPIRE contains four independent retrieval corpora—DailyTalk, VCTK, Expresso, and Synthetic Data—targeting distinct retrieval dimensions.Queries are evaluated only against documents within their own subset, with metrics computed separately.
  • DailyTalk: DailyTalk uses the first dialogue half as a spoken query and the second as its target, testing retrieval of the true conversational continuation.Unrelated dialogues provide negative samples.
  • VCTK: VCTK uses 80 speakers to test whether instructions retrieve utterances from the same speaker as a spoken reference query.The subset uses hard negatives with identical content but different speakers.
  • Expresso: Expresso tests speaker and style constraints across five styles, including tasks matching speaker alone, style alone, or both simultaneously.Same-sentence documents are excluded from positives to prevent trivial exact-match retrieval.
  • Synthetic Data: Synthetic Data samples 200 spoken queries and defines relevance through semantic content, speaker identity, speaking style, environmental sounds, or their combinations.It uses five voices, three expressive styles, and 15 ESC-50 environmental sound effects under controlled conditions.

4. Baseline Methods

The section defines four baseline retrieval paradigms that differ in modality usage and instruction incorporation, each scoring document relevance given a spoken query, instruction, and document. It also evaluates rerankers that reorder top-K first-stage candidates using newly assigned relevance scores.

  • Baseline design: Four baselines vary in modality usage and natural-language instruction incorporation while computing relevance from spoken query q, instruction z, and document d.Each baseline computes f(d | q, z).
  • LALM baseline: Large audio-language models encode task-specific instructions and speech into latent query and document representations, then score relevance with cosine similarity.The query uses the spoken query and instruction, while the document prompt asks for a one-word summary.
  • Cascaded pipeline: The cascaded pipeline transcribes and captions speech, appends the instruction to query text, and applies sparse or dense text retrieval.Queries use [tq; cq; z], whereas documents use [td; cd].
  • Speech-only baseline: The self-supervised speech baseline encodes queries and documents without conditioning on instruction z, making its cosine-based relevance score instruction-independent.This approach has no mechanism to condition retrieval on the given instruction.
  • Contrastive audio-language baseline: The contrastive audio-language baseline concatenates query transcription, caption, and instruction for text encoding, but its encoders are not trained to ground z in the joint space.It preserves acoustic information in document representations and measures zero-shot performance.
  • Reranking: Rerankers re-score and reorder the top-K documents from a first-stage retriever, including LALM-based methods that use the instruction, spoken query, and candidate document.These methods follow LLM- and MLLM-based reranking approaches from other modalities.

5. Experiments

Experiments compare four retrieval paradigms and reranking methods on INSPIRE using recall as the primary retrieval metric and NDCG for reranking. Results show distinct strengths across semantic, acoustic, and multi-attribute retrieval, while reranking generally improves first-stage performance.

  • Experimental Setup: The experiments evaluate large audio-language models, cascaded pipelines, self-supervised speech models, contrastive audio-language embeddings, and multiple rerankers.LALM retrievers include six models, while rerankers use smaller LALMs or text models because they score query-document pairs individually.
  • Evaluation Metrics: Recall is the primary retrieval metric, while reranking is evaluated with NDCG, which gives greater credit to relevant documents at higher ranks.Table 4 reports Recall@10/50/100, and Table 5 reports NDCG@10/50 on the top 100 retrieved documents.
  • Main Retrieval Results: On DailyTalk, cascaded pipelines perform best for semantic conversational-continuity retrieval, with Qwen3-Embedding strongest and E5-Mistral close behind.LALMs remain competitive, whereas contrastive audio-language models underperform because of weak semantic alignment between text and speech.
  • Main Retrieval Results: On VCTK and Expresso, HuBERT-Large and WavLM-Large outperform most alternatives for speaker identity and speaking-style retrieval.These subsets emphasize paralinguistic features, expressiveness, prosody, emotion, and speaker traits that are better captured acoustically.
  • Main Retrieval Results: The Synthetic subset is especially challenging because it combines semantic content, speaker characteristics, speaking style, and environmental sounds.Self-supervised speech and contrastive audio-language embeddings perform very poorly, while LALMs and cascaded pipelines perform better but remain relatively limited.
  • Reranking Results: LALM reranking generally outperforms first-stage LALM retrieval, while text reranking substantially improves DailyTalk and is comparable on other subsets.The paper attributes this contrast partly to excessive information from concatenated transcriptions and captions, which may mislead rerankers.

6. Ablations and Analysis · 6.1. Analysis of Instruction Types · 6.2. Impact of Instruction Usage

Ablations show strong modality specialization: cascaded methods favor semantic constraints, while self-supervised speech models better capture acoustic attributes but struggle with specified values. Instruction benefits remain limited, appearing mainly for instruction-aware text retrievers on DailyTalk and not for paralinguistic or multi-attribute subsets.

  • 6.1. Analysis of Instruction Types: Figure 2 compares four retrieval paradigms by instruction type, normalizing each radar axis by the best-performing model for that attribute.The paradigms are LALMs, cascaded pipelines, self-supervised speech models, and contrastive audio-language models.
  • 6.1. Analysis of Instruction Types: Cascaded methods perform best on semantically driven instructions, including semantic continuation and answer containment, substantially outperforming random baselines and LALMs.The results support specialized text embeddings for precise semantic correspondence in retrieval.
  • 6.1. Analysis of Instruction Types: Self-supervised speech models excel at matching speaker identity, speaking style, and acoustic environment, while cascaded methods and LALMs often perform near chance.This gap indicates insufficient disentangling of paralinguistic cues from linguistic content in current textcentric and unified multimodal architectures.
  • 6.1. Analysis of Instruction Types: Self-supervised speech models perform notably worse when instructions specify particular attribute values, revealing weak fine-grained control.The passage suggests targeted supervision or attribute-aware training strategies as potential requirements.
  • 6.2. Impact of Instruction Usage: Figure 3 compares retrieval with and without instructions across four subsets, using solid bars for instructions and hatched bars without them.Models A–F are LALMs, whereas G–J are cascaded pipelines.
  • 6.2. Impact of Instruction Usage: Instruction gains appear only for instruction-aware text retrievers such as E5-Mistral and Qwen3-Embedding; BM25, SentenceBERT, and six LALMs show minimal sensitivity.BM25 and SentenceBERT are instruction-agnostic, while LALMs lack instruction-aware retrieval training.
  • 6.2. Impact of Instruction Usage: On DailyTalk, instruction-aware text retrievers gain substantially, whereas VCTK, Expresso, and Synthetic show minimal condition differences across models.These results indicate that current approaches cannot leverage instructions for paralinguistic and multi-attribute retrieval, making instruction-aware training necessary.

6.3. Comparison of Captioning Models · 6.4. Effect of Oracle Metadata

Captioning-model comparisons isolate caption quality from retriever choice, finding nearly identical results for detailed and instruction-aware prompts and a sharp performance floor on VCTK and Expresso. Oracle metadata generally helps attribute-focused retrieval but can degrade semantic retrieval by introducing non-semantic noise.

  • 6.3. Comparison of Captioning Models: Captioning comparisons vary only the captioner while holding the downstream text retriever fixed, isolating caption quality from retriever choice.Other LALMs receive detailed prompts because Qwen3-Omni-Captioner does not accept text input.
  • 6.3. Comparison of Captioning Models: Detailed and instruction-aware prompts yield nearly identical results across most configurations.
  • 6.3. Comparison of Captioning Models: All evaluated methods exhibit a sharp performance floor on VCTK and Expresso, regardless of the LALM or prompting strategy.The passage attributes this pattern to likely misalignment between speech features and the retrieval objective.
  • 6.4. Effect of Oracle Metadata: Oracle metadata replaces model-generated captions with ground-truth transcriptions, speaker IDs, speaking styles, and environmental sound labels.The resulting cascaded text-retrieval performance is summarized in Table 6.
  • 6.4. Effect of Oracle Metadata: Oracle metadata degrades semantic retrieval on DailyTalk because non-semantic attributes introduce noise that misleads text matching.
  • 6.4. Effect of Oracle Metadata: On VCTK and Expresso, oracle-metadata effects are mixed but often positive, with VCTK BM25 benefiting dramatically from distinctive oracle speaker-ID tokens.
  • 6.4. Effect of Oracle Metadata: Oracle labels help instruction-aware embeddings when retrieval targets the corresponding attributes directly.
  • 6.4. Effect of Oracle Metadata: On Synthetic, oracle metadata improves most models.

6.5. Comparison with Proprietary Models · 6.6. LALMs Pooling Strategy Comparison

Proprietary cascaded pipelines are competitive for semantic retrieval but remain limited by paralinguistic constraints and the loss of fine-grained acoustic cues. For LALMs, prompt-based last-token embeddings generally outperform mean pooling on content-focused datasets, although instruction-aware training remains necessary.

  • 6.5. Comparison with Proprietary Models: Proprietary APIs use cascaded pipelines that caption speech with a multimodal model and embed the resulting text with a proprietary text model.Google uses Gemini-3.0-Flash with gemini-embedding-001, while OpenAI uses GPT-4o-mini-Audio with text-embedding-3-large.
  • 6.5. Comparison with Proprietary Models: Proprietary models are competitive on semantic retrieval but do not consistently outperform the strongest open-source instruction-aware text retrievers.On DailyTalk, both trail the strongest instruction-aware embedding model; on VCTK and Expresso, both underperform the best open-source baselines.
  • 6.5. Comparison with Proprietary Models: Caption-then-embed pipelines can lose fine-grained acoustic cues by compressing rich speech into text.The limitation motivates instruction-aware speech representations or multimodal embeddings that directly encode acoustic attributes.
  • 6.6. LALMs Pooling Strategy Comparison: The LALM comparison contrasts prompt-based last-token embeddings with mean pooling over all hidden states without a summarization prompt.The prompt-based method appends a summarization prompt and extracts the final token’s hidden state, whereas mean pooling directly averages sequence hidden states.
  • 6.6. LALMs Pooling Strategy Comparison: Prompt-based last-token embeddings generally outperform mean pooling on DailyTalk and Synthetic, particularly for Qwen-series models.Qwen3-Omni shows the largest performance gap, with substantially higher recall for semantic retrieval tasks using the last-token embedding.
  • 6.6. LALMs Pooling Strategy Comparison: On VCTK and Expresso, the two LALM pooling strategies produce similar results.The passage contrasts these datasets with the stronger content-retrieval gains observed on DailyTalk and Synthetic.
  • 6.6. LALMs Pooling Strategy Comparison: The prompt-based last-token strategy is more suitable for LALM retrieval, but overall performance remains limited without instruction-aware retrieval training.The summarization prompt is suggested to guide more discriminative representations for content-based retrieval.

6.7. Layer-wise Analysis of Self-Supervised Speech Models · 6.8. Analysis for Contrastive Audio-Language Embeddings

Layer-wise analysis shows that self-supervised speech models encode different retrieval attributes at different depths, with no single layer optimal across intents. Contrastive audio-language embeddings favor same-modality retrieval, while cross-modal configurations consistently underperform.

  • 6.7. Layer-wise Analysis of Self-Supervised Speech Models: Layer-wise retrieval performance is evaluated for HuBERT-Large and WavLM-Large using representations extracted from different network layers.Figure 5 reports the resulting layer-wise performance analysis.
  • 6.7. Layer-wise Analysis of Self-Supervised Speech Models: VCTK performance peaks in lower-to-middle layers for both HuBERT and WavLM, indicating that speaker information is encoded relatively early.Performance declines in deeper layers after these peaks.
  • 6.7. Layer-wise Analysis of Self-Supervised Speech Models: DailyTalk peaks at specific intermediate-to-higher layers, while Expresso follows a similar but lower trend reflecting balanced speaker and style representations.DailyTalk primarily reveals clearer semantic information in later layers, whereas Expresso balances speaker and style information across layers.
  • 6.7. Layer-wise Analysis of Self-Supervised Speech Models: Synthetic has the lowest recall across all layers for both models, showing that complex instructions combining semantics, speaker, style, and environmental factors remain difficult.Different layers capture different information types, and no single layer is optimal for every retrieval intent.
  • 6.8. Analysis for Contrastive Audio-Language Embeddings: CLAP is analyzed under four encoder configurations that vary the query and document modalities, with A and T denoting audio and text.Configurations using the text encoder first convert spoken utterances to text through ASR and audio captioning.
  • 6.8. Analysis for Contrastive Audio-Language Embeddings: Audio-to-audio retrieval outperforms other configurations on VCTK, Expresso, and Synthetic, whereas text-to-text retrieval performs best on DailyTalk.Audio retrieval captures acoustic attributes, while DailyTalk emphasizes semantic understanding.
  • 6.8. Analysis for Contrastive Audio-Language Embeddings: Cross-modal retrieval consistently yields low recall across all datasets and underperforms both single-modality alternatives.This indicates limited cross-modal alignment for instruction-aware speech retrieval.

7. Conclusion

INSPIRE introduces the first benchmark for instruction-aware speech retrieval, where natural-language instructions specify relevance across multiple attributes. Its evaluation reveals a modality specialization gap between semantic matching and acoustic-property modeling.

  • INSPIRE is the first benchmark for instruction-aware speech retrieval, formalizing natural-language instructions that describe relevance across semantic, speaker, style, and environmental attributes.
  • Cascaded pipelines excel at semantic matching but struggle with paralinguistic attributes.
  • Self-supervised speech models capture acoustic properties but lack instruction sensitivity.

8. Generative AI Use Disclosure

Generative AI tools were used only to polish the manuscript, while the study’s design, data analysis, and scientific interpretations remained the authors’ responsibility and independent of AI influence.

  • Generative AI Use Disclosure: Generative AI tools were used exclusively for manuscript polishing, not for study design, data analysis, or scientific interpretation.The authors retain full responsibility for these aspects, which remain independent of AI influence.
Loading 2608.16203v1…