SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Tasks

1 Introduction

• SLU has received less attention and resources than ASR and speaker identification, and freely available SLU benchmarks are comparatively scarce.Source: §1 · p. 1
• Pipeline systems combine ASR with a text model, whereas end-to-end systems map speech directly to the target output; each approach has distinct advantages.Pipelines can use strong ASR and NLP models; end-to-end models can avoid ASR error propagation and use signal information such as prosody.Source: §1 · p. 1
• Existing SLUE includes only sentiment analysis and named entity recognition, while recent pre-trained models make relatively small task-specific fine-tuning sets feasible.Source: §1 · pp. 1–2
• SLUE Phase-2 adds dialog act classification, question answering, summarization, and named entity localization, including longer discourse inputs and speech-span timestamps.The dataset uses conversational or long-form natural speech rather than synthesized speech or short commands.Source: §1 · p. 2
• The work provides pipeline and end-to-end baselines and analyzes how ASR error rates affect pipeline task performance using multiple ASR systems.Source: §1 · p. 2

2 Related work

• Several existing benchmarks focus on short speech commands, synthesized speech, or test-only evaluation, limiting their fit for broader natural-speech SLU comparisons.Examples include SUPERB, SLURP, ASR-GLUE, and spoken QA datasets.Source: §2 · p. 2
• Speech summarization resources include How-2 and TED Talks, but their raw audio is not publicly available; ICSI and AMI have relatively little annotated data.Source: §2 · p. 2
• Audio de-identification resembles named entity localization but is limited to personal identifiers, and cited Switchboard and Fisher datasets are not free.Source: §2 · p. 3
• Other-language speech efforts exist, though this related-work discussion focuses on English speech.Source: §2 · p. 3

3 SLUE Phase-2: Tasks and data

• SLUE Phase-2 evaluates tasks individually and spans utterance-level classification, longer-context QA and summarization, and named-entity localization.The suite extends SLUE Phase-1 with more diverse and complex tasks.Source: §3 · p. 3, §3.1 · p. 3

3.2.1 SLUE-HVB for DAC

• DAC is utterance-level multi-label classification scored by macro-F1; SLUE-HVB adds human-annotated acts to scripted banking dialogs simulated by 59 speakers.The new act set builds on Switchboard NXT, with labels consolidated based on a pilot annotation.Source: §3.1.1 · p. 3, §3.2.1 · p. 3

3.2.2 SLUE-SQA-5 for QA

• QA predicts answer spans in spoken documents; SLUE-SQA-5 pairs real spoken questions with spoken documents and includes 408 human-verified test pairs.Questions come from five text QA datasets, and the spoken documents are drawn from Spoken Wikipedia.Source: §3.1.2 · p. 3, §3.2.2 · p. 4

3.2.3 SLUE-TED for SUMM

• SUMM requires long-context abstractive speech summaries; SLUE-TED uses public TED audio, with 66% of title words and 57.4% of abstract words present in transcripts.The corpus is split into 80% fine-tune, 10% validation, and 10% test sets.Source: §3.1.3 · p. 3, §3.2.3 · p. 4

3.2.4 SLUE-VoxPopuli for NEL

• NEL localizes named entities by start and end times; SLUE-VoxPopuli adds word-level timestamps to prior NER annotations, verified on 188 dev utterances.NEL annotations are not published for the fine-tune set; the authors focus on repurposing NER models.Source: §3.1.4 · p. 3, §3.2.4 · p. 5

4.1 DAC

• DAC E2E uses wav2vec2 with sigmoid outputs for 18 dialog-act classes, while pipeline baselines pair ASR with DeBERTa classification.Pipeline ASR is either off-the-shelf or fine-tuned on DAC data.Source: §4.1 · p. 5, §4 · p. 5
• DAC pipeline WER and Macro F1 are strongly correlated (Pearson r = -0.9), while ASR fine-tuning adds little over the best NeMo model.Figure 1a shows the relationship across pipeline ASR models.Source: §4.1 · p. 5, Figure 1 · p. 6

4 Experiments and results

• QA pipelines predict answer-span start and end words from ASR transcripts, while E2E-DUAL predicts answer-span indices from speech-derived cluster IDs.Experiments use task fine-tuning sets, select models on dev, evaluate on test, and also measure an oracle pipeline using ground-truth transcripts.Source: §4.1 · pp. 5–6, §4 · p. 5
• QA pipeline WER and frame-F1 are strongly correlated (r = -0.89, p-value < 0.01), and the oracle significantly outperforms all baselines, especially on verified-test.The fine-tuned ASR pipeline-w2v2 does not outperform pipeline-nemo on QA.Source: §4.1 · pp. 6–7, Table 6 · p. 7

4.3 SUMM

• Summarization pipelines pair ASR with LED-large text summarization models, while the E2E system is pretrained for speech recognition before summarization fine-tuning.The pipeline compares summarizers trained on ground-truth and ASR transcripts.Source: §4.3 · p. 7
• Pipeline summarization systems outperform the end-to-end model on ROUGE and METEOR, while the end-to-end model scores higher on BERTScore; all baselines trail the oracle.WER and ROUGE-L are strongly correlated (Pearson r = -0.9, p-value < 0.01); only 6.6% of reference-summary entities appear in predicted summaries.Source: §4.3 · p. 7, Table 7 · p. 8

4.4 NEL

• NEL baselines reuse NER models: CTC-based wav2vec2 models provide frame-level outputs, and pipeline text NER uses DeBERTa.Pipeline entity timestamps come from ASR timing outputs; E2E timestamps come from entity-boundary characters, with offset and trailing-blank heuristics for possible misalignment.Source: §4.4 · pp. 7–8
• For NEL, the lowest-WER models also have the best frame-F1, but overall WER correlation is weak; within model classes, the metrics correlate more strongly.The authors suggest model architecture and/or training objective significantly affects alignment quality.Source: §4.4 · p. 8, Table 8 · p. 8

5 Discussion

• Pipeline models generally outperform end-to-end counterparts among the baselines.The authors note that prior work suggests end-to-end models often have more room for improvement through modeling ideas (Arora et al., 2022a; Pasad et al., 2021).Source: §5 · p. 8
• ASR WER matters more for DAC and summarization, fine-tuned ASR matters for DAC but not QA, and architecture is critical for NEL.The authors state that different pipeline strategies are needed by task.Source: §5 · pp. 8–9

6 Conclusion

• SLUE Phase-2 contributes four additional SLU tasks with high-quality annotations, multiple baselines, and performance analyses using modern ASR and text models.Source: §6 · p. 9
• The suite enables more comprehensive analysis of diverse SLU tasks than previously possible.Source: §6 · p. 9
• Baselines remain far from perfect and relative model performance differs across tasks, leaving room for further SLU research and analysis.Source: §6 · p. 9

• Human performance on the new tasks is unmeasured, although the authors expect it to be better than baseline performance.The authors leave measuring it to future work.Source: Limitations · p. 9
• How much each task benefits from access to audio in addition to text remains unknown and is left for future work.Source: Limitations · p. 9

Broader Impact and Ethics

• SLU benchmarks may support technologies useful to people unable to read or write and to unwritten languages where speech is the only communication form.The authors also hope the work spurs collaboration between speech and NLP researchers.Source: Broader Impact and Ethics · p. 9
• SLUE-SQA workers earned an average of US$10 per hour, while recordings over 30% WER could still be accepted after manual review when content was not substantially missing.The authors also report that DAC annotations and the verified SQA test set were labeled by experienced ASAPP employees.Source: Broader Impact and Ethics · p. 9

A.1 Dialog act list

• Table 9 lists dialog-act details, while the dev-set results section reports WER/F1 analysis and Table 10 experiments.Source: Table 9 · p. 14, §A.1 · p. 14

A.2 Annotation detail

• DAC annotators could assign multiple acts, replay speech, and annotate chronologically combined agent and caller channels; five annotators handled 40 conversation shards.Each conversation was assigned to one annotator, and demographic or geographic information was not collected.Source: §A.2 · p. 15

A.3 Model training details

• DAC E2E models were trained with five random seeds, and the reported model was the median; training used one RTX GPU for about two hours.Pipeline wav2vec2 ASR and DeBERTa NLP fine-tuning took 10 and 3 hours, respectively, on the same GPU.Source: §A.3 · p. 15

A.4 Additional results

• The dev-set DAC WER/F1 plot follows the same trend as the test-set plot, and Table 10 reports baseline performance on dev and test.Source: §A.4 · p. 15, Table 10 · p. 15, Figure 4 · p. 16

B.1 Spoken question collection details

• SLUE-SQA questions were collected through a website that used Google Speech-to-Text to prompt re-recording above 30% WER, but recordings were retained when manual review found no substantial content missing.Workers were paid one dollar to read 50 questions.Source: §B.1 · p. 16

B.2 Search criteria of SLUE-SQA-5 documents

• Paired documents were assessed using BM25 rank, semantic-search rank, and word-F1 from three QA models; dev and test additionally required at least one nonzero QA-model score.Documents also had to contain the answer string and meet the stated BM25 and semantic-search rank thresholds.Source: §B.2 · pp. 16–17
• Questions without a document meeting all criteria were discarded; when multiple documents qualified, the highest-ranked semantic-search result was paired, with one document per question.Source: §B.2 · p. 17

B.3 Model training details

• The E2E-DUAL model converts wav2vec2 representations to k-means cluster IDs for Longformer answer-span prediction, while the pipeline fine-tunes wav2vec2 ASR and DeBERTa NLP models.Source: §B.3 · p. 17

B.4 Additional results

• Table 12 reports dev-set QA baseline performance and identifies stt-en-contextnet-1024 as the best NeMo model by frame-F1.Source: Table 12 · p. 18
• Dev-set document WER has a strong negative correlation with QA frame-F1 (Pearson r=-0.94, p<0.01), while question WER correlates more weakly on both dev and test.Source: §B.4 · p. 17, Figure 6 · p. 18, Figure 7 · p. 19

C.1 Model details

• ASR inference on TED talks used independent 10-second chunks, while NeMo conformer and squeezeformer inference required five-minute chunks.Source: §C.1 · p. 20
• The E2E summarization model used only the first 30,000 speech frames (600 seconds) because full TED talks did not fit on a GPU.It used only the SSL model's last hidden state.Source: §C.1 · p. 20

C.2 Additional dataset details

• SLUE-TED contains longer audios and transcripts than prior works, with dataset statistics summarized in Table 13.Source: §C.2 · p. 20, Table 13 · p. 20

C.3 Additional results

• Dev-set WER and ROUGE-L show a correlation trend similar to the test set.Figure 10 plots the dev-set WER and ROUGE-L scores.Source: §C.3 · p. 20, Figure 10 · p. 22
• Pipeline summaries produce more accurate words, while E2E summaries are more semantically similar to references; both differ from the reference summaries.The E2E model scores slightly better on titles than abstracts: ROUGE-L 15.2 and BERTScore 87.7 versus ROUGE-L 14.4 and BERTScore 83.4.Source: §C.3 · p. 20, Table 16 · p. 23

D.1 Annotation details

• MFA failed on 26 files: transcript endpoint errors explained 22, seven files were skipped for missing entity phrases, and four received manual alignments.For a verified sample of 188 utterances, rectified alignments had 96% frame-F1 against the original, and baseline-score relative differences were under 3%.Source: §D.1 · p. 24

D.2 Hyperparameter details

• NEL tuning varied offset from -0.3 to 0.3 seconds in 20-millisecond steps and used a Boolean incl_blank setting; the relevant settings differed by NeMo decoding strategy.For RNNT models, offset was the only relevant hyperparameter; for CTC models, incl_blank governs whether specified blank tokens are included in entity timestamps.Source: §D.2 · p. 24

D.3 Error analysis

• Within a single NeMo model category, NEL and ASR performance are correlated.Figure 12 plots WER against F1 scores for NeMo models.Source: §D.3 · p. 26
• The E2E NEL model has higher precision, while the pipeline model has higher recall; text-NER training on gold text but inference on ASR output could increase pipeline false positives.The table reports dev-set baseline precision and recall.Source: §D.3 · p. 24, Table 18 · p. 26

D.4 Additional results

• Relaxing word-F1 tolerance from ρ = 1 to ρ = 0.8 boosts performance by up to 30% for pipeline models and 116% for E2E models.Table 19 reports dev- and test-set performance across word-F1 thresholds.Source: Table 19 · p. 26, §D.4 · p. 26

E Experiment detail

• The experiment details list the NeMo models and their parameter sizes; base-sized models are used when variants exist, except off-the-shelf ASR models.Figure 12 shows test-set WER and frame-F1 for different NeMo models.Source: E Experiment detail · p. 27, Figure 12 · p. 27, Table 20 · p. 28

D.4 Additional results

• Dev-set WER and frame-F1 show a correlation trend similar to the test set.Figure 13 plots the dev-set WER and frame-F1 scores.Source: §D.4 · p. 26, Figure 13 · p. 30