SpeechDPR: End-to-End Spoken Passage Retrieval for Open-Domain Spoken Question Answering

1. INTRODUCTION

• Low-resource languages motivate openSQA models that require no paired speech-text data, while UASR errors and OOV words challenge cascaded retrieval and question answering.The paper says inaccurate UASR transcripts can cause downstream errors and may omit named entities or OOV words.Source: §1 · p. 1
• Earlier end-to-end spoken SQA methods found answers in given passages, while prior spoken retrieval addressed query-by-example phrase detection or visually grounded search rather than openSQA sentence-level semantics.The openSQA retrieval question may not share overlapping content with its gold passage.Source: §1 · p. 1
• SpeechDPR is an end-to-end model for untranscribed spoken-question-to-spoken-passage semantic retrieval, distilling sentence-level representations from a cascading UASR and text dense retriever.It uses a bi-encoder and scores question-passage similarity with the inner product of their sentence representations.Source: §1 · pp. 1–2
• SpeechDPR achieves competitive retrieval accuracy against cascading baselines and significantly outperforms them when UASR performs poorly.The paper notes that spoken questions and passages may include named entities and OOV words that UASR cannot recognize.Source: §1 · pp. 1–2

2. BACKGROUND

• OpenSQA retrieval ranks spoken archive passages against a spoken question and returns a small set for downstream SQA to locate the answer span.A retrieval counts as successful when the returned set includes the gold passage; the setting assumes neither paired speech-text data nor off-the-shelf supervised ASR.Source: §2 · p. 2

2.1. Bi-encoder dense retriever framework

• The bi-encoder separately encodes questions and passages into vectors, scores their similarity by dot product, and learns representations by minimizing negative log likelihood against irrelevant passages.In this paper, the negative passages are other questions’ gold passages from the same batch.Source: §2.1 · p. 2

3.1. SpeechDPR model

• SpeechDPR processes waveforms with a self-supervised speech encoder, feature processor, and separate question and passage sentence encoders.The speech encoder is frozen; a feature processor normalizes and downsamples its representations before Roberta-base encoders produce 768-dimensional sentence vectors.Source: §3.1 · p. 2

3.2. Knowledge distillation from Cascading Teacher model

• The Cascading Teacher combines UASR trained on unpaired speech and text with a text retriever trained on UASR transcripts rather than ground-truth transcripts.SpeechDPR adds teacher distillation because preliminary training with only the NLL objective performed incompetently.Source: §3.2 · p. 2

3.3. Training objective

• SpeechDPR’s objective combines its retrieval NLL with two cross-model NLL terms: student-question to teacher-passage and teacher-question to student-passage.The three losses are weighted by tuned hyperparameters α and β.Source: §3.3 · pp. 2–3

4.1. Data

• SLUE-SQA-5 contains about 46k training, 1.9k dev, and 2.4k test questions, with distinct speakers across subsets and 40-second paired passages containing answers.Questions are human-spoken and drawn from five TQA datasets; experiments pair them with Spoken Wikipedia passages.Source: §4.1 · p. 3
• The Spoken Wikipedia archive comprises about 39k 40-second passages from 1.2k articles, totaling 427 hours; experiments are limited to English for lack of suitable datasets in other languages.Source: §4.1 · p. 3

4.2. Evaluation

• Retrieval is evaluated by top-20 accuracy, while openSQA passes the top 20 passages to a shared SQA module and measures answer spans with FF1.A correct answer from a passage not labeled gold receives FF1 zero, which may underestimate model performance.Source: §4.2 · p. 3

4.3. Implementation details

• The cascading teacher uses wav2vec-U 2.0 for UASR and trains a Roberta-base bi-encoder TDR on UASR transcriptions.UASR uses unpaired speech and text for adversarial training, then speech for self-training.Source: §4.3 · p. 3
• SpeechDPR uses HuBERT-large representations from layer 22, with its feature processor and two sentence encoders trained jointly; checkpoints are selected by dev-set top-20 accuracy.The reported setup uses a feature processor with strides 4 and 3 in its two CNN layers, and sets α and β to 0.5.Source: §4.3 · p. 3
• The shared openSQA evaluator combines UASR with a Deberta-base TQA model and selects the answer span using interpolated passage-similarity and span scores.The TQA model is fine-tuned on SQuAD v1.1, and the evaluator obtains 11.17 FF1 on the test set when given each question's gold passage.Source: §4.3 · p. 3

4.4. Baseline: the cascading approach (UASR + TDR)

• With no known prior work for this openSQA retrieval task, the study uses UASR plus TDR as its baseline and also trains a distilled cascading student.Source: §4.4 · p. 3

5.1. Retrieval results

• 28.88% vs. 25.60%: the SpeechDPR–Cascading Teacher ensemble exceeds the two-cascading-model ensemble in Top-20 retrieval accuracy.The authors attribute the gain to complementary knowledge learned by the end-to-end SpeechDPR model.Source: Table 1 · p. 4, §5.1 · p. 4

5.2. OpenSQA results

• 0.951 vs. 0.860: the SpeechDPR–Cascading Teacher ensemble exceeds the two-cascading-model ensemble in openSQA FF1.The openSQA score is evaluated by passing the top-20 retrieved passages to a shared SQA model.Source: Table 1 · p. 4, §5.2 · p. 4

5.1. Retrieval results

• 19.73%: SpeechDPR’s Top-20 retrieval accuracy is comparable to Cascading Teacher at 19.94% and Cascading Student at 19.90%.The reported differences are less than 1%.Source: Table 1 · p. 4, §5.1 · p. 4
• 0.04%: SpeechDPR without knowledge distillation has substantially lower retrieval accuracy, indicating that distillation from the cascading model is crucial.The ablation removes knowledge distillation from the Cascading Teacher or the specified distillation loss terms.Source: §5.1 · p. 4

5.2. OpenSQA results

• OpenSQA accuracy is constrained by the downstream SQA model; on SLUE-SQA-5, the best model without paired speech data achieves only 21.8 FF1.The paper cites the SLUE-SQA-5 paper for the 21.8 FF1 result [27].Source: §5.2 · p. 4

5.3. Retrieval performance at different WERs

• When UASR WER exceeds 40%, SpeechDPR significantly outperforms the cascading baseline; its retrieval is more robust as WER rises.The figure plots Top-20 retrieval accuracy across test-question subsets grouped by UASR WER.Source: §5.3 · p. 4, Figure 2 · p. 4

• SpeechDPR uses knowledge distillation from a cascading UASR and text dense retriever model for openSQA retrieval without supervised ASR or manual transcripts.Source: §6 · p. 4
• SpeechDPR performs competitively against the cascading baseline and significantly better in scenarios with poor UASR performance.Source: §6 · p. 4
• Initial experiments were conducted only in English; extending the framework to other low-resource languages is left to future research.Source: §6 · p. 4