Catching the Drift: Probabilistic Content Models, with Applications to Generation and Summarization

Motivation

• A distributional approach can reduce human effort and identify topics that human experts might not anticipate, provided recurrent patterns exist in domain texts.Texts from the same domain are described as sufficiently similar for such patterns to be learned.Source: §1 · p. 1
• Content models represent domain-specific topics and the order in which information about them is presented.The models target content rather than domain-independent rhetorical elements.Source: §1 · p. 1

Method

• The method learns topics and topic relations directly from unannotated documents using a knowledge-lean adaptation of standard HMM induction.This avoids manually determining the topics for each domain.Source: §1 · p. 1

Caveat

• Formulaic text need not be simple, so automated methods remain advantageous when modeling several domains.This qualifies the assumption that recurrent domain patterns support distributional learning.Source: §1 · p. 1

Applications

• 78 percentage points separated content models from the state-of-the-art ordering method in one domain and metric, while summarization matched human-written output 88% versus 69% for the leading n-sentences baseline.These experiments cover information ordering and extractive summarization.Source: §1 · pp. 1–2

Conclusion

• The paper argues that content models are flexible and effective enough to represent important text properties and may apply beyond the two studied tasks.The authors identify broader applications as future research rather than an evaluated result.Source: §1 · p. 2

Knowledge-rich approaches

• Content models use distributional evidence to incorporate domain, domain-independent communication, and domain communication knowledge when those patterns recur in texts.This contrasts with formalisms that typically target one knowledge category through manual representations.Source: §2 · p. 2
• Although content models are relatively impoverished representations, their simplicity facilitates learning and their empirical results show strong effectiveness.The related-work discussion presents simplicity as an advantage for learnability rather than as a complete representation of content.Source: §2 · p. 2

Knowledge-lean approaches

• Unlike Duboue and McKeown’s method, this approach applies when no domain-specific knowledge base has been supplied.The comparison concerns required resources, not the methods’ relative performance.Source: §2 · p. 2
• Content-model induction is closely related to distributional methods used in text segmentation and topic-based language modeling.The paper situates its learning techniques within that existing literature.Source: §2 · p. 2
• Unlike topic-based language-modeling work, this paper treats content as the primary modeling target and therefore adapts HMM induction beyond standard Baum-Welch learning.Language-modeling methods generally use topic as an auxiliary variable for predicting text probabilities.Source: §2 · p. 2

Determining states, emission probabilities, and transition probabilities

• An iterative re-estimation procedure alternates between clustering spans by word distributions and estimating state language models and topic-transition models.The procedure is designed to learn both topics and their relations directly from unannotated documents.Source: §3 · p. 2

Formalism preliminaries

• Content models are HMMs whose states represent domain topics, generate topic-relevant sentences, and encode topic shifts through transition probabilities.The model assumes texts in a domain share one content model and supports forward and Viterbi inference.Source: §3 · p. 2

Initial topic induction

• The method initializes topics by clustering sentences with bigram-based cosine similarity, then merges small clusters into an “etcetera” insertion cluster.Proper names, numbers, and dates are temporarily replaced with generic tokens to encourage event-type rather than event-identity clustering.Source: §3 · p. 3

Determining states, emission probabilities, and transition probabilities

• State emissions use smoothed cluster-derived bigram probabilities, while transitions are estimated from how sentences from the same articles move between clusters.The insertion state is forced to model digressions or unseen topics by making its language model complementary to the others.Source: §3 · p. 3

Viterbi re-estimation

• Viterbi re-estimation reassigns sentences to the states most likely to have generated them, repeating clustering and parameter estimation until stabilization or an iteration limit.This incorporates contextual ordering information that initial lexical clustering ignores.Source: §3 · p. 3

4 Evaluation Tasks

• Content models support two complementary tasks: ordering pre-selected information for text synthesis and selecting representative sentence subsequences for summarization.The tasks test distinct model capabilities: topic ordering and topic representation for selection.Source: §4 · p. 3

4.1 Information Ordering

• For information ordering, the content model selects the highest-probability sequence among candidate orderings for a domain.The task is motivated by applications including concept-to-text generation and multi-document summarization.Source: §4.1 · p. 4

4.2 Extractive Summarization

• The summarization algorithm combines unannotated learning with global topical structure rather than relying on manually specified content rules or mainly local sentence features.It uses document-summary pairs while operating over the topical structure of the entire document.Source: §4.2 · p. 4
• For each state, summary-generation probability is estimated from document-summary pairs after Viterbi assigns topic labels to article and summary sentences.Only states appearing in at least three full training articles are used for this probability computation.Source: §4.2 · p. 4
• To create a length-ℓ summary, the method assigns V-topics to article sentences and selects the ℓ states with the highest learned summary-generation probabilities.It outputs the article sentences corresponding to those selected states.Source: §4.2 · p. 4

5.1 Data

• Content models were evaluated on five domains, with 100 training articles, 100 test articles, and 20 development articles per domain.The domains were earthquakes, clashes, drug-related offenses, financial reports, and aviation accidents.Source: §5.1 · p. 4

5.2 Parameter Estimation

• Four free parameters were tuned separately by domain on held-out development data, producing models with 32 to 95 states and sharp language models.Parameter values were selected to optimize information-ordering performance.Source: §5.2 · pp. 4–5

5.3.1 Metrics

• OSO prediction rate and Kendall’s τ complemented OSO rank because Lapata’s method did not consider all sentence permutations.OSO rank compares the original sentence order with all permutations, while Kendall’s τ measures disagreement with that order.Source: §5.3.1 · p. 5

5.3.2 Results

• Content models outperformed a bigram baseline and Lapata’s ordering method almost universally, with Earthquakes placing the original order among the top five permutations in 95% of documents.The authors attribute the advantage to modeling global document structure rather than only local word or sentence relationships; performance improved with training-set size.Source: §5.3.2 · p. 5, Table 2 · p. 6

5.4 Summarization Experiments

• The summarization model outperformed both the leading-sentences baseline and a sentence-level classifier, reaching 88% human-summary sentence match versus 69% for the leading-sentences baseline.Evaluation used 60 Earthquakes document-summary pairs, split evenly into training and testing, with extraction accuracy based on overlap with human-selected sentences.Source: §5.4 · p. 6, Figure 3 · p. 7, Table 4

5.5 Relation Between Ordering and Summarization Methods

• On Earthquakes, model-size rankings were similar for ordering and summarization, and the same-size model performed best on both tasks despite the comparison covering only one domain.This correlation supports using content models as general representations of text structure, while the authors note the domain limitation.Source: §5.5 · p. 7, Table 5 · p. 7

• Content models yield substantial improvement over previously proposed methods in ordering and summarization applications.Source: §6 · p. 7
• The method captures domain-specific constraints on both topic selection and topic organization without annotated training data.Source: §6 · p. 7
• A future direction is connecting domain-specific content models with domain-independent formalisms such as RST.Source: §6 · p. 7
• The authors also plan to investigate hierarchical models bootstrapped from content-model labels and guided by domain-independent discourse constraints.Source: §6 · p. 7