Source-linked AI summary
LibriSpeech-PC: Benchmark for Evaluation of Punctuation and Capitalization Capabilities of end-to-end ASR Models
Aleksandr Meister, Matvei Novikov, Nikolay Karpov, Evelina Bakhturina, Vitaly Lavrukhin, Boris Ginsburg
TL;DR
ASR punctuation and capitalization prediction faces limited suitable data and evaluation metrics that do not isolate punctuation errors. This paper introduces the LibriSpeech-PC dataset and PER for evaluating punctuation and capitalization prediction; its reported findings favor the end-to-end Conformer model over the cascade model.
Problem
End-to-end ASR with punctuation and capitalization faces a lack of suitable datasets, while standard metrics do not distinguish punctuation errors from other token mismatches.
Source: §1 · p. 1Method
The paper presents a benchmark combining a punctuation-and-capitalization-restored LibriSpeech dataset with PER, an edit-distance metric focused on punctuation marks.
Source: §3 · pp. 2–3Results
The end-to-end Conformer-based ASR model trained on LibriSpeech-PC shows superior performance to the cascade model.
Source: §7 · p. 6Takeaways & Limitations
PER provides punctuation-focused evaluation independent of words, including granular assessment of individual punctuation marks.
Source: §7 · p. 6For F1-based comparison, intersecting zero-WER samples can make evaluation sets smaller and metrics non-reproducible because intersection sizes vary with model quality.
Source: §6.1 · p. 5
Abstract
from arXiv · showhide
Traditional automatic speech recognition (ASR) models output lower-cased words without punctuation marks, which reduces readability and necessitates a subsequent text processing model to convert ASR transcripts into a proper format. Simultaneously, the development of end-to-end ASR models capable of predicting punctuation and capitalization presents several challenges, primarily due to limited data availability and shortcomings in the existing evaluation methods, such as inadequate assessment of punctuation prediction. In this paper, we introduce a LibriSpeech-PC benchmark designed to assess the punctuation and capitalization prediction capabilities of end-to-end ASR models. The benchmark includes a LibriSpeech-PC dataset with restored punctuation and capitalization, a novel evaluation metric called Punctuation Error Rate (PER) that focuses on punctuation marks, and initial baseline models. All code, data, and models are publicly available.
1. INTRODUCTION
The introduction identifies a lack of suitable punctuation-and-capitalization training data and metrics that separate punctuation and capitalization errors from word errors. It presents LibriSpeech-PC, a restored dataset, the PER metric, and initial cascade and end-to-end baselines.
1. INTRODUCTION
- Existing WER, CER, and F1 metrics treat ASR token mismatches alike, rather than distinguishing word, punctuation, and capitalization errors.This leaves punctuation and capitalization errors unassessed as separate categories.Source: §1 · p. 1
- The paper introduces LibriSpeech-PC, a benchmark for evaluating joint ASR models with punctuation and capitalization.Its components include a punctuation-focused metric, a restored dataset, and initial cascade and end-to-end baselines.Source: §1 · p. 1
- The paper introduces Punctuation Error Rate (PER) to evaluate end-to-end ASR models with punctuation and capitalization.PER enables separate analysis of punctuation and capitalization errors.Source: §1 · p. 1
- LibriSpeech-PC adapts LibriSpeech by restoring punctuation and capitalization.Source: §1 · p. 1
- The benchmark includes initial English cascade and end-to-end ASR models with punctuation and capitalization.Source: §1 · p. 1
2. RELATED WORK
Prior work explores punctuation and capitalization prediction with linguistic, prosodic, audio, and text features, using both separate and end-to-end models. The section also reviews evaluation approaches and motivates a new benchmark metric.
2.1. Punctuation and Capitalization Prediction
- Earlier approaches combined linguistic and prosodic features in decision trees and multilayer perceptrons, then audio and text features in an LSTM.Source: §2.1 · pp. 1–2
- Cascade and transformer approaches restore punctuation and capitalization from ASR text [8], but require additional maintenance and increase pipeline latency.Source: §2.1 · p. 2
- Later work introduced end-to-end acoustic prediction [11] and compared end-to-end and cascade approaches [12], including a separate punctuation decoder.Source: §2.1 · p. 2
2.2. Metrics for Punctuation and Capitalization
- F1-based evaluation can be inaccurate for ASR when tokens are absent from either the prediction or ground truth, especially with recognition errors.Source: §2.2 · p. 2
- The paper introduces a new benchmark metric to address challenges in meaningful comparison and benchmarking.The PER calculation masks punctuation tokens, finds a shortest path, then checks predicted punctuation at correct positions.Source: §2.2 · p. 2, Figure 1 · p. 2
3. PROPOSED METRIC
The proposed PER adapts edit-distance evaluation to a predefined target subset—punctuation marks—rather than the full word sequence. It assesses punctuation-position and token correctness, with normalization and scope distinct from WER.
3. PROPOSED METRIC
- PER is based on Levenshtein distance, whose elementary operations also underlie WER.The operations include substitutions, deletions, insertions, and matches.Source: §3 · p. 2
- PER evaluates a predefined subset of tokens—punctuation marks—while WER aggregates edit operations over all tokens.The metric is intended to compare joint end-to-end ASR models and cascades on punctuation prediction.Source: §3 · pp. 2–3
- PER calculates punctuation errors from insertions, deletions, and substitutions relative to those errors plus correctly predicted punctuation tokens.The operation counts apply to punctuation tokens only.Source: §3 · p. 3
- The PER alignment masks target tokens before computing the backtrace, locating punctuation positions before checking exact token matches.This defines substitutions within the target group rather than using the naive algorithm substitution operation.Source: §3 · p. 3
- PER normalizes total errors by errors plus correct predictions, yielding values from 0 to 1 interpretable like accuracy.Source: §3 · p. 3
- PER supports granular evaluation of a model's prediction of a specific token subset, unlike WER's general-purpose assessment of overall ASR performance.Source: §3 · p. 3
4. LIBRISPEECH-PC DATASET
LibriSpeech-PC restores punctuation and capitalization in LibriSpeech text by tracing audiobook text sources and aligning them with existing transcripts. It preserves the original sample segmentation and subset divisions while dropping samples that meet specified exclusion conditions.
4. LIBRISPEECH-PC DATASET
- LibriSpeech-PC restores punctuation and capitalization by tracing audiobook text sources and aligning them with LibriSpeech transcripts.The original sample segmentation and subset divisions are preserved.Source: §4 · p. 3
- Alignment exclusions include samples with non-standard Unicode mismatches, more than two words of deviation, or duration under one second.Source: §4 · p. 3
- Samples entirely in uppercase were also dropped because this typically reflected an error in the original book text.Source: §4 · pp. 3–4
- Each split retained more than 90% of its original duration after punctuation-and-capitalization restoration.Table 1 compares LibriSpeech and LibriSpeech-PC duration.Source: §4 · p. 4, Table 1 · p. 4
5. EXPERIMENTS
The experiments compare cascade and end-to-end ASR approaches for punctuation and capitalization, using LibriSpeech-PC transcripts prepared with a restricted punctuation set. They also test whether n-gram language models built from LibriSpeech-PC, Common Crawl, or both can improve end-to-end decoding.
5. EXPERIMENTS
- Only periods, commas, and question marks are retained; punctuation is separated from words by spaces and repeated whitespace is collapsed.Other punctuation is removed because it is limited in the dataset.Source: §5 · p. 4
- The experiments compare end-to-end ASR models that predict punctuation and capitalization with a cascade that adds a text-to-text punctuation model after ASR.The cascade uses a Conformer-based ASR model followed by a BERT model with separate punctuation and capitalization heads.Source: §5 · p. 4, §5.1 · p. 4
5.1. Base and Tuned Cascades
- The tuned cascade fine-tunes the pretrained BERT punctuation and capitalization model for three epochs.Training uses batches of 15,000 tokens per GPU, Adam, and a learning rate of 1×10−4.Source: §5.1 · p. 4
5.2. E2E ASR with Punctuation and Capitalization
- The Conformer end-to-end model is fine-tuned to predict punctuation and capitalization from a pretrained checkpoint for 100,000 steps.It uses a SentencePiece Unigram tokenizer with a vocabulary of 128 tokens.Source: §5.2 · p. 4
5.3. E2E ASR with n-gram Language Models
- The n-gram language models use word-piece BPE tokens matching the ASR token set, with 7-gram variants trained on LibriSpeech-PC, Common Crawl, or an equal-weight mix.The Common Crawl subset is 100 times larger than LibriSpeech-PC and is normalized and cleaned similarly; combining models uses equal weights.Source: §5.3 · p. 4
- Beam-search decoding with a language model performs better than the pure end-to-end model.The experiments use OpenGrm NGram to combine two n-gram models with equal weights.Source: §5.3 · p. 4
6. RESULTS
The results compare cascade and end-to-end models using F1-based measures and word-error-rate variations. On test-clean, the end-to-end model with a joint LibriSpeech-PC and Common Crawl language model has the best reported overall performance, while punctuation-focused examples show how PER captures errors that WER-based measures may understate.
6.1. F1-based Metrics
- The E2E model outperforms the Tuned Cascade on the reported F1, precision, and recall comparisons for capitalization and punctuation.These comparisons use only samples with WER=0, which make up 48–66% of test-clean and test-other samples in the two setups.Source: §6.1 · pp. 4–5
- 375 test-clean samples (15.5%) and 1,119 test-other samples (39.1%) remain in the intersection of zero-WER subsets, and these sizes can vary with model quality, making the metrics not reproducible.A fair comparison requires this intersection, which could further reduce the evaluation set.Source: §6.1 · p. 5
6.2. WER-based Metrics
- 7.66% WER PC and 29.24% PER: the E2E model with the joint LibriSpeech-PC and Common Crawl language model has the best overall test-clean performance.The E2E model without a language model reports 8.11% WER PC and 29.48% PER; the Tuned Cascade reports 9.45% WER PC and 36.6% PER.Source: §6.2 · p. 5, Table 3 · p. 5
- 100.0% PER versus 25.0% WER PC: PER captures the complete absence of punctuation in “I was done .” when the hypothesis omits the period.The example illustrates why WER PC alone may not make this punctuation failure apparent.Source: §6.2 · p. 5
- 50.0% PER versus 20.0% WER PC: PER captures the missing comma in “Let’s eat , Bob !” when the hypothesis reads “Let’s eat Bob !”.The passage notes that this punctuation error is important to conveying intended meaning.Source: §6.2 · p. 5
- PER isolates punctuation-mark errors, whereas WER and CER account for errors across all tokens and are minimally effective for evaluating punctuation errors.Figure 2 compares the metrics’ sensitivity to different punctuation errors.Source: §6.2 · p. 6, Figure 2 · p. 6
7. CONCLUSIONS
The paper introduces PER and the LibriSpeech-PC dataset for evaluating punctuation and capitalization recovery in ASR. Its empirical findings report superior performance for the end-to-end Conformer model compared with the cascade model.
7. CONCLUSIONS
- PER operates independently of words and provides a granular evaluation of individual punctuation marks, including commas, periods, and question marks.The paper presents PER as an alternative to established metrics such as F1 and WER for punctuation and capitalization recovery in ASR.Source: §7 · p. 6
- The end-to-end Conformer-based ASR model trained on LibriSpeech-PC shows superior performance to the cascade model.LibriSpeech-PC adapts LibriSpeech with restored punctuation and capitalization.Source: §7 · p. 6, Table 4 · p. 6
- The paper argues for including PER as a standard evaluative metric in ASR.It presents this as a conclusion about ASR evaluation and future progress in the field.Source: §7 · p. 6