Source-linked AI summary
A Reranker for Orchestrating Heterogeneous Speech and Text Retrievers
Inho Kim, Sumyeong Ahn
TL;DR
RAG research has limited support for heterogeneous speech-and-text databases and cross-modal candidate comparison. STeReO builds cross-modal ranking data and reranks candidates from modality-specific retrievers, improving reranking and downstream question answering in single- and mixed-modality settings.
Problem
RAG systems have limited support for heterogeneous speech-and-text databases, whose candidates require cross-modal relevance comparison and dedicated training data.
Method
STeReO constructs cross-modal relevance-ranking data and trains a reranker to unify and prioritize candidates from independent speech and text retrievers.
Results
STeReO consistently improves reranking and downstream QA performance over the Z-score baseline in both single-domain and mixed-modality settings.
Takeaways & Limitations
The reranker provides more relevant context for question answering across speech, text, and mixed-modality evidence pools.
Takeaways & Limitations
For text-only candidates, the annotation pipeline uses gpt-4o because gpt-4o-audio-preview does not support text-only queries.
Abstract
from arXiv · showhide
Retrieval-Augmented Generation (RAG) systems have attracted significant interest for their ability to mitigate hallucinations in Large Language Models (LLMs). Although knowledge databases for RAG are increasingly diversifying to include various modalities such as speech and text, research on handling such multi-modal database scenarios remains limited. In this paper, we propose STeReO (Speech and Text Reranking Orchestrator), a reranker based on speech and text retrievers that aggregates disparate modality databases. To address the lack of specialized training data, we first curate a dataset comprising queries, mixed-modality evidence, and their corresponding relevance ranks. We then train the reranker and evaluate its effectiveness in both single-modality and mixed-modality scenarios. Our results demonstrate that the proposed algorithm excels at selecting the most relevant evidence, thereby significantly improving downstream question-answering performance.
1 Korea Institute of Energy Technology, South Korea · 1. Introduction
The introduction motivates mixed-modality RAG by identifying limitations in speech and text retrieval, then presents STeReO as a cross-modal reranker trained on explicitly ranked heterogeneous evidence. It also outlines a benchmark evaluation showing improvements over single-modality baselines.
- 1. Introduction: RAG incorporates external knowledge into LLMs to mitigate hallucinations and ground responses in evidence, but most pipelines assume text-only databases.The paper identifies a growing need to integrate unstructured spoken content such as lectures and meeting recordings.
- 1. Introduction: ASR-free methods avoid some speech-recognition limitations, yet VoxRAG and SpeechRAG remain restricted to speech-only retrieval spaces.WavRAG instead projects independent audio and text databases into a shared embedding space, but heterogeneous retrieval can suffer systematic score imbalance.
- 1. Introduction: Late fusion merges candidates from modality-specific retrievers, making a robust cross-modal reranker essential for prioritizing evidence from diverse modalities.The introduction contrasts this need with listwise reranking methods previously demonstrated mainly on text-only candidate pools.
- 1. Introduction: STeReO is a cross-modal reranker that aligns and prioritizes candidates from independent modality-specific databases into one ranked list for a text query.Its training begins with a dataset explicitly capturing cross-modal relevance rankings among heterogeneous-retriever candidates.
- 1. Introduction: The proposed dataset provides explicit relevance rankings for heterogeneous candidates and uses score fusion to merge disjoint retriever outputs into a targeted evidence subset.This filtering enables extraction of relevance orders from the broader evidence space.
- 1. Introduction: The dataset supports optimization with pointwise, pairwise, and listwise reranking objectives, while evaluation uses a cross-modal benchmark built from Spoken SQuAD and MS MARCO.Experiments cover various backbone architectures and compare against existing single-modality baselines.
2. The proposed method STeReO
STeReO orchestrates modality-specific speech and text retrieval by reranking their combined candidates for Audio Language Model question answering. Its method constructs cross-modal relevance-labeled data, then trains a parameter-efficient reranker with flexible ranking objectives.
- Phase 1: Dataset Construction: Phase 1 aggregates candidates from independent speech and text retrievers and assigns unified relevance labels with a foundation ALM.The dataset is built from cross-modal candidate collections annotated using a foundation ALM such as GPT.
- Framework: STeReO uses modality-specific retrievers to generate candidate sets, whose union is reranked into mixed evidence for an Audio Language Model.For query q, each retriever produces C_m, the reranker forms C⋆ from their union, and the ALM generates the answer from C⋆ and q.
- Phase 1: Dataset Construction: Z-normalization aligns modality-specific retrieval scores before candidates are merged and the top-k samples are selected for unified labeling.Raw scores are normalized using each modality’s mean and standard deviation, producing a unified labeling set L.
- Phase 2: Reranker Training: Phase 2 fine-tunes decoder-based Audio Language Models as rerankers on cross-modal candidate-relevance tuples using Low-Rank Adaptation.The stated reranker architectures include Ultravox, Qwen-Audio-Chat, and Qwen2-Audio.
- Phase 2: Reranker Training: The training pipeline supports Pointwise binary crossentropy, Pairwise RankNet-style, and Listwise ApproxNDCG objectives.The framework is designed to integrate any of these ranking losses, with σ(·) denoting the sigmoid and τ set to 10 in the listed objectives.
3. Experiment
The experiment evaluates STeReO on heterogeneous speech-text and text-only retrieval datasets using fixed top-5 retrieval, multiple ranking objectives, and single- versus mixed-domain scenarios. Results assess reranking, downstream QA, fusion, pooling, and audio-window configurations, with findings supporting both label quality and retrieval effectiveness.
- Experimental Setup: STeReO uses a fixed top-k pipeline with k = 5 on Spoken SQuAD and MS MARCO, combining approximately 2.8K audio and 9.1K text passages.Spoken SQuAD contains text queries and TTS-generated audio passages, whereas MS MARCO contains text-only web passages.
- Experimental Setup: The foundation ALM produces labels with F1 = 0.700 and MCC = 0.614 against passage-ID ground truth on a held-out set of 5,000 candidates.Text-only candidates use gpt-4o because gpt-4o-audio-preview does not support text-only queries.
- Evaluation: Evaluation compares Single-domain reranking, which removes cross-domain noise, with Mixed-domain reranking over the full heterogeneous pool.Reranking uses Hit@1, MRR, and NDCG@5, while downstream QA additionally reports substring-matching Exact Match.
- Main Results: In the Single setting, all three backbones outperform the Z-score baseline within their native domains when optimized with the appropriate objective.The reported objectives are Pointwise, Pairwise, and Listwise.
- Fusion Analysis: 0.527 Hit@1 is recovered by Z-score normalization on Spoken SQuAD, while unnormalized fusion and RRF suffer near-zero Hit@1 (≈0.05) from modality collapse.Raw text scores overwhelm speech candidates without effective normalization.
- Audio Analysis: 0.758 Hit@1 is achieved by 30s×4 windowing on Spoken SQuAD with Qwen2-Audio, outperforming a single 120s×1 window under a fixed 120-second budget.Max pooling also consistently outperforms Mean pooling on Spoken SQuAD, where speech candidates occupy a larger share of the top-5 pool.
4. Conclusion
STeReO effectively reranks heterogeneous speech-and-text pools, maintaining high single-domain precision and robust discrimination in mixed-modality scenarios. Modality-wise Z-score normalization and optimized audio windowing address structural challenges and support consistent performance gains.
- 4. Conclusion: STeReO maintains high precision in single-domain speech or text environments.
- 4. Conclusion: STeReO ensures robust discrimination when speech and text evidence are combined.
- 4. Conclusion: Modality-wise Z-score normalization and optimized audio windowing address structural challenges, including sequence length constraints.
- 4. Conclusion: The proposed strategies achieve consistent performance gains across the evaluated settings.