Source-linked AI summary
Text Segmentation as a Supervised Learning Task
Omri Koshorek, Adir Cohen, Noam Mor, Michael Rotman, Jonathan Berant
TL;DR
Text segmentation lacked large labeled natural-text datasets, leaving prior work dependent on unsupervised methods. The paper formulates segmentation as supervised sentence labeling, introduces WIKI-727K from Wikipedia, and develops a hierarchical neural model. The model outperforms prior methods on Wikipedia, performs competitively on earlier benchmarks, and runs in linear time with modern GPU hardware.
Problem
Existing text-segmentation datasets are small, and the absence of a large labeled dataset led prior work to use unsupervised or heuristic methods.
Method
The paper automatically constructs WIKI-727K from Wikipedia and models segmentation as sentence-level supervised labeling with a hierarchical LSTM architecture.
Results
The model outperforms prior methods on Wikipedia documents, performs competitively on prior benchmarks, and has linear runtime in text length.
Takeaways & Limitations
WIKI-727K provides a path toward training and evaluating segmentation systems on arbitrary natural text.
Takeaways & Limitations
Segmentation results are not always directly comparable because some prior methods assume datasets whose documents share a topic.
Abstract
from arXiv · showhide
Text segmentation, the task of dividing a document into contiguous segments based on its semantic structure, is a longstanding challenge in language understanding. Previous work on text segmentation focused on unsupervised methods such as clustering or graph search, due to the paucity in labeled data. In this work, we formulate text segmentation as a supervised learning problem, and present a large new dataset for text segmentation that is automatically extracted and labeled from Wikipedia. Moreover, we develop a segmentation model based on this dataset and show that it generalizes well to unseen natural text.
1 Introduction
Text segmentation divides documents into topically coherent segments whose boundaries indicate topic changes, supporting downstream applications such as summarization and information extraction. This work addresses limited labeled data by formulating segmentation as supervised learning and introducing WIKI-727K, a large automatically labeled Wikipedia dataset, alongside a hierarchical neural model.
- Text segmentation divides documents into topically coherent segments, with cutoff points marking topic changes.The resulting structure can support downstream applications such as summarization and information extraction.
- Existing datasets are small, often evaluation-focused, and sometimes synthetic, while no large labeled dataset was available.Prior approaches therefore relied on heuristics for sentence-topic similarity or explicit topic models such as LDA.
- The paper formulates text segmentation as supervised learning, labeling each sentence according to whether it ends a segment.
- WIKI-727K contains more than 727,000 English Wikipedia documents automatically segmented using each document’s table of contents.The dataset is intended to provide a large, natural, broad-coverage benchmark and is publicly released with the code.
- A hierarchical neural model uses a bidirectional LSTM for sentence representations and a higher-level LSTM to label sentences, outperforming prior methods.
2 Related Work
Earlier text-segmentation benchmarks were small and included synthetic or narrowly scoped datasets, limiting the availability of labeled natural text. Prior methods used Bayesian topic models or unsupervised graph-based heuristics to identify segment boundaries.
- Existing datasets: The Choi dataset contains 920 synthetic documents formed by concatenating 10 random Brown-corpus passages.
- Existing datasets: Glavaš et al. created a dataset of five manually segmented political manifestos from the Manifesto project.
- Existing datasets: Chen et al. evaluated segmentation on two small English Wikipedia datasets containing 100 city documents and 118 chemical-element documents.
- Existing datasets: Existing datasets were small and could not benefit from supervised training over labeled data.
- Prior methods: Bayesian methods represented documents with sampled topics and vocabulary distributions, then segmented using coherence drops between adjacent sentences.
- Prior methods: GRAPHSEG builds a sentence-similarity graph and determines segments through maximal cliques of adjacent sentences plus heuristic completion.
3 The WIKI-727K Dataset
WIKI-727K is a large, hierarchically segmented English Wikipedia corpus designed to provide natural, open-domain training and evaluation data. Its complete hierarchy supports different application-specific levels of segmentation granularity.
- Dataset construction: WIKI-727K contains 727,746 English Wikipedia documents with hierarchical segmentation derived from their tables of contents.Documents were randomly split into 80% training, 10% development, and 10% test sets.
- Segmentation hierarchy: The dataset preserves multiple segmentation granularities, allowing applications to select top-level segments or finer sub-segments.Examples include broad topics such as History and Geography versus separate news stories in a broadcast.
- Preprocessing: Preprocessing removed photos, tables, templates, and other non-text elements, then filtered unsuitable documents and single-sentence segments.
- Preprocessing: Each retained segment was divided into sentences with the PUNKT tokenizer to establish a well-defined benchmark unit.
- Dataset motivation: The corpus is intended to be natural, open-domain, and easy to expand because additional training data can be collected at little cost.
4 Neural Model for Text Segmentation
The model hierarchically encodes words into sentence representations, then predicts whether each sentence ends a segment. It is trained end-to-end with cross-entropy and uses thresholded greedy decoding at test time.
- The supervised formulation represents a document as n sentences and labels each of the first n−1 sentences according to whether it ends a segment.
- A two-layer bidirectional LSTM at the lower level consumes each sentence's word tokens and max-pools its outputs into a sentence representation.
- A higher-level two-layer bidirectional LSTM consumes the sentence embeddings and produces segmentation predictions through a fully connected layer and softmax.
- 4.1 Training: The model predicts a cutoff probability for each relevant sentence and minimizes summed cross-entropy errors over the n−1 prediction targets.
- Training uses end-to-end stochastic gradient descent, while the system predicts only top-level segmentation after removing Wikipedia's first summary segment during training.
- At test time, greedy decoding creates a segment whenever a cutoff probability exceeds a validation-optimized threshold τ.
5 Experimental Details
The experiments evaluate the model on WIKI-727, synthetic Choi data, and small Wikipedia benchmarks using Pk, runtime, and human-performance comparisons. Results show stronger performance on natural Wikipedia text, competitive transfer to prior benchmarks, and substantially better runtime characteristics than GRAPHSEG, while cross-dataset comparisons have limitations.
- Evaluation setup: The model is evaluated on WIKI-727, Choi’s synthetic dataset, and the CITIES and ELEMENTS Wikipedia datasets.WIKI-50 provides a 50-document sample for evaluating computationally slower systems and measuring human segmentation performance.
- Evaluation setup: Pk measures boundary-classification error using a sliding window whose size is half the average ground-truth segment size.Word-based Pk is additionally reported for the datasets from Chen et al. (2009).
- Experimental limitations: Segmentation results are not always directly comparable because Chen et al. (2009) require all documents in a dataset to discuss the same topic.The paper nevertheless attempts a comparison in Table 2.
- Accuracy: GRAPHSEG performs better on the synthetic Choi dataset, but underperforms the random baseline on natural Wikipedia data.The synthetic dataset concatenates unrelated documents, making topic changes easier to detect through word-level similarity than within a natural document.
- Accuracy: The method outperforms Chen et al. (2009) on CITIES but performs worse on ELEMENTS, possibly because its Google News embeddings contain fewer chemistry-specific terms.The authors describe the result as competitive despite not exploiting the shared document structure or training specifically for these datasets.
- Accuracy: Human performance on WIKI-50 is only slightly better than the model’s performance.The authors hypothesize that annotators’ limited exposure to documents leaves them less familiar with the appropriate segmentation granularity.
- Run Time: The method has runtime linear in document word and sentence counts, whereas GRAPHSEG has asymptotic complexity O(N^3+V k).Neural models are also highly parallelizable and can benefit from GPU execution; runtime is reported per document on a CPU in Table 3.
6 Conclusions
The paper presents WIKI-727K as a large labeled dataset enabling supervised neural text segmentation, and reports strong performance with efficient execution. Future work targets richer sentence-level models and structured global decisions.
- WIKI-727K enables training neural text segmentation models with supervised learning, addressing the literature’s prior reliance on unsupervised training.
- The model outperforms prior methods on Wikipedia documents and performs competitively on prior benchmarks.
- The system has linear runtime in text length and can run on modern GPU hardware.
- Future work will explore richer neural models at the sentence level.
- A structured global model is proposed as a future direction to incorporate all local predictions into a global segmentation decision.