Spoken SQuAD: A Study of Mitigating the Impact of Speech Recognition Errors on Listening Comprehension

1. Introduction

• Prior spoken-language work included sequence-labeling models and ASR confidence features, while SQA commonly used ASR transcripts with retrieval or knowledge bases.The related TOEFL listening-comprehension study used a limited dataset and multi-select answers rather than machine-produced answers.Source: §1 · p. 1
• Spoken content is less investigated than text comprehension, despite the difficulty and time required for people to access large spoken or multimedia collections.Source: §1 · p. 1
• Spoken SQuAD is an extraction-based spoken-question-answering task with a new evaluation approach for spoken documents and text questions.The system can return either a text answer or an audio time span containing the answer.Source: §1 · p. 1, Figure 1 · p. 1
• ASR errors significantly degrade QA, with state-of-the-art SQuAD models showing worse performance on Spoken SQuAD transcriptions.The authors describe the impact as catastrophic.Source: §1 · p. 1
• The paper proposes several approaches to mitigate the impact of ASR errors on QA.Source: §1 · p. 1

2.1. Corpus Description

• The dataset uses spoken versions of SQuAD articles with text questions, and removes question-answer pairs whose answers are absent from the ASR transcript.It contains 37,111 training and 5,351 testing pairs; the training and testing WERs are 22.77% and 22.73%, respectively.Source: §2.1 · pp. 1–2
• Three test sets cover different WER levels, created by adding two levels of white noise to synthesized test audio.The authors note that synthesized speech makes the task easier than real application audio and leave real recordings for future work.Source: §2.1 · p. 2, Table 1 · p. 2

2.2. Evaluation Metrics

• Text answers are evaluated with Exact Match and F1, which compare the predicted answer string with the ground-truth answer.Exact Match is 1 only for an exact match; F1 uses word-level precision and recall.Source: §2.2 · p. 2
• The audio-span evaluation is motivated by cases where comprehension identifies the correct answer location but ASR misrecognizes the answer words.The authors argue that selecting the correct audio segment better evaluates SQA reasoning than requiring an exact transcript string.Source: §2.2 · p. 2
• Audio answers are evaluated by Audio Overlapping Score, the overlap between predicted and ground-truth answer time spans.AOS rewards overlap and penalizes predicted spans that are too long.Source: §2.2 · p. 2

3. Model

• The SQA system combines an ASR module with a reading-comprehension module.Source: §3 · p. 2

3.1. Reading Comprehension Models

• The evaluated reading-comprehension systems were selected from competitive models with state-of-the-art SQuAD results; BiDAF [26] is one listed model.Source: §3.1 · p. 2
• At test time, models extract answer spans from machine transcripts; experiments compare SQuAD versus ASR-transcript training and SQuAD-dev versus Spoken SQuAD test performance.Source: §3.1 · p. 2

3.2. Subword Units

• Phoneme-CNN applies stride-1 filters over phoneme embeddings, max-pools each filter’s n-gram features, and concatenates the scalars into a word representation learned end-to-end with the reader.The same CNN architecture can also use syllable sequences.Source: §3.2 · p. 3

4. Experiments Setup and Results

• The experiments cover published SQuAD models, text-versus-transcript training, subword-embedding ablations, and a qualitative prediction example.The paper reports that phoneme and syllable embeddings improve BiDAF on test sets with different WERs.Source: §4 · p. 3

4.1. Investigating the Impact of ASR Errors

• 75.42% average F1 on text fell to 55.4% with ASR errors across five models, with a similar pattern in EM.The models were trained on the full SQuAD training set; Table 2 compares SQuAD-dev and SpokenS-test.Source: §4.1 · p. 3, Table 2 · p. 3

4.2. Training on ASR Transcriptions

• The SQuAD training set was almost two times larger, yet ASR-trained models were remarkably better on ASR-error test data.Documents whose correct answers were absent from ASR transcriptions could not be used to train BiDAF and Dr.QA.Source: Table 3 · p. 3, §4.2 · p. 4, §4.3 · p. 4

4.3. Mitigating ASR errors by Subword Units

• With BiDAF trained on SQuAD, dropout improved results, while phoneme or syllable sequence embeddings improved them further.The proposed phoneme and syllable embeddings are produced from unit sequences using Phoneme-CNN and concatenated with other word representations.Source: Table 4 · p. 3, §4.3 · p. 4
• Row (e), combining character, phoneme, and syllable sequence embeddings, achieves the best EM/F1/AOS performance across all WER conditions.The comparison is on the Spoken SQuAD testing set.Source: §4.3 · p. 4, Table 5 · p. 4

4.4. Qualitative Analysis

• In the example, ASR misrecognizes “area” as “harry”; word-only BiDAF is confused, while phoneme-embedding BiDAF selects the correct answer span.The question begins with “Who,” prompting the word-only model to seek a person’s name near “Los Angeles county.”Source: §4.4 · p. 4, Table 6 · p. 4

• Spoken SQuAD is an extraction-based spoken question-answering task, and the paper reports ASR-related performance degradation and proposes subword units as mitigation.Source: §5 · p. 4