Source-linked AI summary
Disfluency Detection using a Bidirectional LSTM
Vicky Zayats, Mari Ostendorf, Hannaneh Hajishirzi
TL;DR
Disfluencies complicate spoken-language processing, and detecting both reparanda and repairs matters for transcript cleanup and interaction analysis. The paper combines BLSTM sequence tagging with pattern-match features and ILP constraints, reporting state-of-the-art results for edit and correction detection.
Problem
Disfluencies challenge spoken-transcript tasks such as parsing and machine translation, while detecting both disfluent and correction parts can matter for transcript processing and interaction analysis.
Source: §1 · p. 1Method
Results
The model achieves 85.9 F-score for standard edit detection and 57.7 versus 49.6 F-score for correction detection compared with the CRF result reported in.
Source: §5.2 · p. 4, §7 · p. 4Takeaways & Limitations
Performance gains include cases reported as hardest to detect, including restarts and repairs.
Source: §7 · p. 4The best BLSTM models rely on hand-crafted pattern-match features, suggesting the architecture does not learn these cues sufficiently with the available annotated training data.
Source: §7 · p. 4
Abstract
from arXiv · showhide
We introduce a new approach for disfluency detection using a Bidirectional Long-Short Term Memory neural network (BLSTM). In addition to the word sequence, the model takes as input pattern match features that were developed to reduce sensitivity to vocabulary size in training, which lead to improved performance over the word sequence alone. The BLSTM takes advantage of explicit repair states in addition to the standard reparandum states. The final output leverages integer linear programming to incorporate constraints of disfluency structure. In experiments on the Switchboard corpus, the model achieves state-of-the-art performance for both the standard disfluency detection task and the correction detection task. Analysis shows that the model has better detection of non-repetition disfluencies, which tend to be much harder to detect.
1. Introduction
Disfluencies distinguish spontaneous speech and complicate NLP tasks such as parsing and machine translation, while also reflecting speaker interaction. This work models both reparanda and repairs, with the longer-term goal of understanding variation in disfluency production.
1. Introduction
- Disfluency structure comprises a reparandum, an interruption point, an associated repair, and an optional interregnum.Ignoring the interregnum, restarts have an empty repair, repetitions repeat the reparandum, and corrections differ from it.Source: §1 · p. 1
- Unlike transcript-cleaning work focused on reparandum detection, this study also targets repairs to support later analysis of cognitive-load and social-context variation.The cited table is described as giving examples of disfluency types and annotations.Source: §1 · p. 1, Table 1 · p. 1
- The approach combines a BLSTM with integer linear programming and achieves state-of-the-art performance on the standard Switchboard disfluency detection task.The authors also report contributions from pattern-match features.Source: §1 · p. 1
2. Related Work
Prior work uses sequence tagging and parsing-based models for disfluency detection, with distinct trade-offs in latency, joint parsing, and annotation requirements. This paper uses a non-incremental recurrent classifier with bidirectional context and end-to-end feature embeddings.
2. Related Work
- Sequence-tagging systems commonly use BIO-style reparandum labels and discriminative models such as CRFs; the current best-performing system is a Semi-Markov CRF.ILP with a CRF and F-score matching, multi-step learning, and M3N are also explored.Source: §2 · p. 1
- Parsing models represent reparandum–repair relations, but noisy-channel models are unsuited to restarts, and parsing requires treebank training annotations.Parsing models can jointly optimize parsing and disfluency detection; incremental dependency models tend to underperform delayed-decision models because they lack reparandum–repair similarity cues.Source: §2 · p. 1
- Unlike incremental RNN detection or RNN embeddings used as CRF features, this study uses a non-incremental bidirectional RNN as the classifier and trains embeddings end-to-end.The model also uses pattern-matching features and leverages reparandum–repair relatedness for repetitions and corrections.Source: §2 · p. 2
- This study focuses on lexical features alone, although prosodic cues such as pause duration and fundamental-frequency trends can help detect interruption points.Prior studies generally find the gain from combining prosodic and lexical features relatively small.Source: §2 · p. 2
3. General Framework
The framework extends sequence-tagging states to represent repairs, repetitions, and corrections while retaining a CRF baseline built from a core feature set. CRF templates expand those core features into a larger set of local features.
3. General Framework
- The 5-state edit-detection scheme is extended to 8 states with explicit repair labels and to 17 states for joint reparandum and correction detection, distinguishing repetition from non-repetition disfluencies.Explicit repair states improve edit detection but do not themselves distinguish repetitions from corrections; the 17-state scheme expands non-outside states for that distinction.Source: §3 · p. 2
- The CRF baseline starts from lexical, distance-based pattern-match, and disfluency-language-model features, which CRF++ templates expand within a relative time window.The resulting set contains 258 features, including combinations and n-grams of core features and POS n-grams.Source: §3 · p. 2
4. Proposed Method
The proposed method uses recurrent sequence tagging with bidirectional context, learned feature embeddings, and ILP post-processing. Its inputs combine words, POS tags, and disfluency cues, while ILP constraints address invalid neighboring labels.
4.1. RNN Architectures
- Bidirectional LSTMs use past and future states to predict each word's disfluency tag, providing context useful for repair and correction prediction.The architecture takes word, POS, and core-feature inputs and produces output states.Source: §4.1 · p. 2, Figure 1 · p. 3
4.2. Feature Embeddings
- Word, POS, and disfluency-based features are mapped separately to embeddings, concatenated as LSTM inputs, and updated by backpropagation during network training.Word embeddings are initialized using a backward-LSTM language model trained on Switchboard with disfluencies removed; POS initialization follows a similar procedure.Source: §4.2 · p. 3
4.3. ILP post-processing
- ILP post-processing addresses illegal label sequences by collapsing larger-state softmax probabilities into five states for the existing constraints.For the 8-state model, probabilities for O, IE, and IP incorporate the corresponding C, C IE, and C IP states.Source: §4.3 · p. 3
5. Experiments
Experiments on Switchboard assess edit detection and correction detection, then compare the best models with prior methods. The evaluation also examines development-set performance and implementation settings.
5. Experiments
- The experiments use Switchboard’s standard training, development, and test split, and evaluate word-level edit regions plus finer-grained edit types and corrections.Models are tuned on the development set; training excludes sentences longer than 50 words, while testing processes all sentences.Source: §5 · p. 3
5.2. Method Comparison
- 85.9 F-score on test edit detection beats the best prior lexical-cue-only result and the prosodic-feature result (F=85.4).The authors report a 7% reduction in the performance gap from 1-F of 15.2 to 14.1.Source: Table 5 · p. 4, §5.2 · p. 4
5.1. Model Performance
- On the development set, 17-state models enable correction detection but give slightly worse edit detection; the BLSTM has significantly better overall correction F-score, with different precision-recall tradeoffs.The authors note that ILP post-processing could potentially recover some precision lost with the 17-state BLSTM.Source: §5.1 · pp. 3–4, Table 4 · p. 4
5.2. Method Comparison
- 57.7 F-score for correction detection exceeds the CRF result reported in (49.6), a 16% improvement.Source: §5.2 · p. 4
6. Analysis
The analysis compares CRF, LSTM, and BLSTM feature settings, then examines performance on repetition and non-repetition disfluencies.
6.1. Ablation study: CRF vs LSTM
- With words alone, LSTM and BLSTM outperform CRF; adding POS and pattern-match features improves all systems, but benefits CRF more.The comparison is for 5-state edit detection on the development set; CRF uses 1–3-gram word features within a window of 8, while LSTM and BLSTM use the current word index and recurrent context.Source: §6.1 · p. 4, Table 6 · p. 4
6.2. Repetitions vs. Non-repetitions
- The BLSTM is much better than the CRF at predicting non-repetitions, which are harder to detect than repetitions.The authors conjecture that dense BLSTM word representations capture reparandum/repair “rough copy” similarities better than simple POS pattern-match features.Source: §6.2 · p. 4, Table 7 · p. 4
7. Conclusion and Future Work
The paper concludes that its BLSTM approach achieves state-of-the-art edit detection and improves correction detection. It also identifies a dependence on hand-crafted pattern-match features given the available annotated training data.
7. Conclusion and Future Work
- The BLSTM reaches 85.9 F-score on the standard disfluency detection task using explicit repair states, lexical feature embeddings, and ILP post-processing.Source: §7 · p. 4
- The paper also reports improved correction detection and gains on the harder-to-detect restarts and repairs.Source: §7 · p. 4
- With the available annotated training data, the best BLSTM models still rely on hand-crafted pattern-match features rather than learning those cues automatically.Whether other neural architectures can learn the cues more effectively and whether the features help across domains remain open questions.Source: §7 · p. 4