Source-linked AI summary
Text Segmentation by Cross Segment Attention
Michal Lukasik, Boris Dadachev, Gonçalo Simões, Kishore Papineni
TL;DR
Document and discourse segmentation break text into units that can support downstream tasks, but prior approaches were not directly comparable across datasets and often did not use transformers or contextual embeddings. The paper introduces three transformer-based architectures and compares them with recent approaches on three datasets. Its models improve the reported state of the art, while cross-segment BERT is competitive with hierarchical models.
Problem
Prior segmentation approaches were evaluated on different datasets and mostly relied on RNNs rather than newer transformers or contextual embeddings.
Source: §1 · p. 2Method
The paper proposes three transformer-based segmentation architectures, including local-context cross-segment BERT and two hierarchical models, and compares them with recent approaches on three datasets.
Source: §1 · p. 2, §3.1 · p. 3, §3.3 · p. 4Results
Takeaways & Limitations
Distillation is an effective way to build much more compact models for practical settings.
Source: §7 · p. 9The authors caution that local context may be insufficient in some applications, and that Brown-corpus splits reuse documents across sets, creating data leakage.
Source: §4.1 · p. 5, §7 · p. 8
Abstract
from arXiv · showhide
Document and discourse segmentation are two fundamental NLP tasks pertaining to breaking up text into constituents, which are commonly used to help downstream tasks such as information retrieval or text summarization. In this work, we propose three transformer-based architectures and provide comprehensive comparisons with previously proposed approaches on three standard datasets. We establish a new state-of-the-art, reducing in particular the error rates by a large margin in all cases. We further analyze model sizes and find that we can build models with many fewer parameters while keeping good performance, thus facilitating real-world applications.
1 Introduction
Text segmentation divides documents into coherent units and discourse into sub-sentence units; both can support downstream tasks. The paper addresses limited cross-study comparability and limited transformer use with new architectures and comparisons across three datasets.
1 Introduction
- Document segmentation breaks text into logically coherent sub-document units, while discourse segmentation divides text into Elementary Discourse Units.The introduction describes applications including information retrieval, summarization, and information extraction for document segmentation, and summarization and discourse analysis for discourse segmentation.Source: §1 · p. 1
- Prior approaches were evaluated on different datasets, limiting direct comparison, and mostly relied on RNNs rather than newer transformers or contextual embeddings.The paper frames these gaps as motivation for its comparative evaluation and transformer-based models.Source: §1 · p. 2
- The authors compare recent segmentation approaches on three public datasets and introduce three transformer-based architectures, reporting a new state of the art.They also analyze architectural strengths and weaknesses, the effect of context size and model size, and whether a simple local-context paradigm remains competitive.Source: §1 · p. 2
2 Literature review
The literature review traces document segmentation from unsupervised methods toward supervised neural approaches, while discourse segmentation has relied on supervised learning and external resources.
Document segmentation
- Unsupervised document segmentation methods use lexical cohesion or Bayesian approaches, but are difficult to specialize by domain and do not naturally handle task-dependent granularity.The review notes that supervised methods have consequently been a focus of recent work.Source: §2 · p. 2
- Neural document-segmentation approaches include hierarchical Bi-LSTM sequence labeling, BiGRU sequence-to-sequence models, and attention-based CNN-Bi-LSTM models.Reported evaluations span Wikipedia, synthetic Brown-corpus documents, and other small-scale datasets.Source: §2 · p. 2
Discourse Segmentation
- Discourse segmentation is typically supervised, but limited data has encouraged reliance on external annotations and resources to support generalization.Earlier feature-based systems depended on linguistic annotations, while later end-to-end neural models used pretrained embeddings instead.Source: §2 · pp. 2–3
3 Architectures
The paper proposes three architectures for segmentation: a local-context BERT model and two models that encode full context hierarchically. The latter use sentence representations followed by either a Bi-LSTM or a transformer encoder.
3 Architectures
- The architectures comprise one local-context model and two full-context hierarchical models; the latter share sentence-level BERT encoding but differ in their upper-level encoder.BERT+Bi-LSTM uses a Bi-LSTM, whereas hierarchical BERT uses a transformer; the proposed models are shown for document segmentation in Figure 3.Source: §3 · p. 3, §3.2 · p. 3, §3.3 · p. 4
3.1 Cross-segment BERT
- Cross-segment BERT represents each candidate break using the local token context on its left and right, then feeds the concatenated contexts to a BERT-initialized transformer.The contexts are separated by [SEP], with [CLS] at the start; each side can contain at most 255 word-pieces.Source: §3.1 · p. 3
3.2 BERT+Bi-LSTM
- BERT+Bi-LSTM independently encodes each sentence, then uses a Bi-LSTM over sentence representations to model sequences of indefinite size.For sentence-level decisions it uses [CLS] representations; for word-level decisions it uses the left-most word-piece representation of each word, with a 64-token sentence limit and documents chunked at 128 sentences.Source: §3.2 · pp. 3–4
3.3 Hierarchical BERT
- Hierarchical BERT encodes sentences independently and passes their [CLS] representations to a transformer document encoder that relates sentences through cross-attention.The model replaces the document-level LSTM with a transformer encoder and uses 64 word-pieces per sentence and 128 sentences per document.Source: §3.3 · p. 4
4 Evaluation methodology
The evaluation covers document segmentation on Wiki-727K and Choi, and discourse segmentation on RST-DT. It reports dataset characteristics and evaluates internal segment boundaries using Precision, Recall, F1-score, and, on Choi, Pk.
4.1 Datasets
- Wiki-727K and Choi evaluate document segmentation, while RST-DT evaluates discourse segmentation; Table 1 summarizes dataset statistics.Source: §4.1 · p. 4, Table 1 · p. 5
- Wiki-727K contains 727,000 English Wikipedia articles and evaluates section boundaries using the authors’ original data splits.Its average document has 3.5 segments, averaging 13.6 sentences per segment; the study reused the original preprocessing script for fair comparison.Source: §4.1 · pp. 4–5
- Choi consists of 700 synthetic documents, each formed from 10 segments sampled from Brown corpus articles.The Brown corpus’s 500 documents were sampled repeatedly across train, validation, and test splits, causing data leakage; the authors discourage future use of this dataset.Source: §4.1 · p. 5
- RST-DT contains 385 Wall Street Journal articles, with 347 for training and 38 for testing; results average 10-fold cross-validation because validation-set choice strongly affects performance.Source: §4.1 · p. 5
4.2 Metrics
- Precision, Recall, and F1-score measure internal segment boundaries, excluding each document’s or sentence’s final boundary.On Choi, the study also reports Pk, where lower is better; the authors note that Pk has known biases, including penalizing false negatives more than false positives.Source: §4.2 · p. 5
5 Results
Across three datasets, the proposed models outperform the baselines, with the largest relative error reduction on Choi. Cross-segment BERT performs well using local context, while model structure and initialization affect results and training cost.
5 Results
- Table 2 compares the proposed models with hierarchical Bi-LSTM, SEGBOT, and BiLSTM+CRF+ELMO, and includes a human annotation baseline for RST-DT.Standard deviations were estimated for proposed models and, where possible, by bootstrapping the hierarchical Bi-LSTM test set.Source: §5 · p. 5, Table 2 · p. 6
- 20%, 16%, and 79% are the relative error-margin reductions over the best baseline on Wiki-727K, RST-DT, and Choi, respectively; improvements are statistically significant on all three.Source: Table 2 · p. 6, §5 · p. 6
- Removing the Bi-LSTM from the RST-DT hierarchical model decreased F1-score by 0.4%, while LSTM-related models were harder to train and significantly slower on Wiki-727K.Source: §5 · p. 6
- HIBERT initialization was necessary for good hierarchical BERT results on Choi, whereas initializing both hierarchy levels with BERTBase gave slightly better Wiki-727K results but took longer to converge.Other tested initializations performed worse.Source: §5 · p. 6
- Cross-segment BERT performs well across datasets using local context; on RST-DT, 255-255 contexts yield 67.1 F1, 73.9 recall, and 61.5 precision.On RST-DT, hierarchical transformer encoding does not improve over cross-segment BERT.Source: §5 · pp. 6–7
6 Analyses
The analyses examine cross-segment BERT’s context needs, model size, and distillation, motivated by its strong performance and simple use of local context. They find that both sides of a candidate break matter, larger models generally perform better, and distillation helps compact models compete.
6 Analyses
- The analyses focus on cross-segment BERT because it is simple, competitive with hierarchical approaches, and uses local context without encoding an entire document.The context-length study considers reducing trailing context from 128 word-piece tokens to zero while holding leading context at 128 tokens.Source: §6 · p. 7, §6.1 · p. 7
6.1 Role of trailing context size
- F1 is 20.2 with 256 tokens before and none after a break, versus 64.0 with 128 tokens on each side; sorted token lists score 39.1 versus 64.0 for running text.These results indicate that both sides of the break matter and that high-performing models use more than token counts to detect semantic shift.Source: §6.1 · p. 7
6.2 Role of Transformer architecture
- Larger models generally perform better, but configuration also matters: a 128-dimensional model with more layers can outperform a 256-dimensional model with fewer layers.The authors report that the state-of-the-art gain came at a steep model-size cost, motivating their exploration of distillation.Source: §6.2 · pp. 7–8
6.3 Model distillation
- Distilled models improve F1-scores by over 4 points over models trained without a teacher, and compact models can outperform the previous state of the art.The authors note that exact model-size comparisons with Koshorek et al. are not possible because their embedding archive contains over 3M vocabulary items.Source: §6.3 · p. 8
7 Conclusion
The paper introduces three transformer-based architectures for document and discourse segmentation and evaluates them on three standard datasets. All improve the state of the art, while cross-segment BERT is competitive with hierarchical models; the authors also report that distillation can produce more compact models for practical settings.
7 Conclusion
- The three proposed architectures improve the current state of the art, and cross-segment BERT is extremely competitive with hierarchical models.The paper describes cross-segment BERT as using only local context around candidate breaks.Source: §7 · p. 8
- Local context is sufficient in many cases, but the results do not imply that hierarchical models should be disregarded.The authors say applications may exist where local context is insufficient.Source: §7 · p. 8
- Deep transformer encoders appear useful for long, complex document inputs, while LSTMs proved useful for discourse segmentation and RNNs may help with very long documents.The authors connect RNN usefulness for very long documents to their ability to handle very long input sequences.Source: §7 · pp. 8–9
- The authors report that distillation builds more compact models for practical settings and plan to investigate data augmentation, regularization, and mitigation of labeling noise.The future-work directions include data augmentation and methods for regularization and mitigating labeling noise.Source: §7 · p. 9