PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Document-Level Semantic Properties from Free-Text Annotations

S. R. K. Branavan, Harr Chen, Jacob Eisenstein, Regina Barzilay

arXiv:1401.3457v1cs.CLcs.IR

TL;DR

Document-level semantic inference needs scalable annotations, but free-text keyphrases can be inconsistent and incomplete. The paper jointly links keyphrase clusters with latent topics in document text, and reports stronger single- and multi-document property summarization than competitive baselines.

  • Problem

    Free-text keyphrase annotations are abundant but inconsistent across paraphrases and incomplete because relevant document properties may be left unlabeled.

    Source: §1 · p. 1

  • Method

    A hierarchical Bayesian model jointly links clusters of free-text keyphrases with latent topics in document text to infer semantic properties.

    Source: §1 · p. 2, §7 · p. 29

  • Results

    The model outperforms competitive baselines for semantic property extraction from single and multiple documents, while joint text modeling yields a better paraphrase structure.

    Source: §1 · p. 3, §7 · p. 29

  • Takeaways & Limitations

    The system can aggregate multiple reviews by comparing inferred properties even when reviewers use different surface forms.

    Source: §2.3 · p. 6, §7 · p. 29

    The model assumes semantic properties are unstructured, leaving relations such as antonymy and hierarchy unmodeled.

    Source: §7 · p. 30

Abstract

from arXiv · show

This paper presents a new method for inferring the semantic properties of documents by leveraging free-text keyphrase annotations. Such annotations are becoming increasingly abundant due to the recent dramatic growth in semi-structured, user-generated online content. One especially relevant domain is product reviews, which are often annotated by their authors with pros/cons keyphrases such as a real bargain or good value. These annotations are representative of the underlying semantic properties; however, unlike expert annotations, they are noisy: lay authors may use different labels to denote the same property, and some labels may be missing. To learn using such noisy annotations, we find a hidden paraphrase structure which clusters the keyphrases. The paraphrase structure is linked with a latent topic model of the review texts, enabling the system to predict the properties of unannotated documents and to effectively aggregate the semantic properties of multiple reviews. Our approach is implemented as a hierarchical Bayesian model with joint inference. We find that joint inference increases the robustness of the keyphrase clustering and encourages the latent topics to correlate with semantically meaningful properties. Multiple evaluations demonstrate that our model substantially outperforms alternative approaches for summarizing single and multiple documents into a set of semantically salient keyphrases.

1. Introduction

Document-level semantic inference needs scalable annotations, but author-written keyphrases are abundant yet inconsistent and incomplete. The paper jointly models keyphrase clusters and text topics, outperforming alternatives on single- and multi-document summarization.

1. Introduction

  • Free-text annotations offer a growing, linguistically intuitive source of training data, but labels vary across synonymous phrases and omit properties present in the document text.For example, reviews may describe the same healthy-food property with different phrases, while a service property goes unannotated.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • The approach uses text regularities to cluster paraphrased keyphrases and estimates semantic topics from document text to address missing annotations.The text and keyphrases are modeled as reflecting one shared set of semantic properties.Source: §1 · p. 2
  • A hierarchical Bayesian model links an LDA-style word-generation component with keyphrase similarity and clusters, while a background topic distribution covers properties absent from annotations.Word topics are drawn from distributions governed by annotated keyphrase clusters and a background distribution.Source: §1 · pp. 2–3
  • The trained system clusters annotation phrases and infers properties in unlabeled reviews; its topic model also abstracts reviews for cross-document comparison and aggregation.The review aggregation system was applied across 480 product categories, covering 49,490 products and 522,879 reviews.Source: §1 · p. 3
  • The model substantially outperforms alternatives for single- and multi-document property summarization, and text-linked clustering improves agreement with expert keyphrase groupings.The authors also find that clustering free-text annotations is essential for extracting meaningful semantic properties.Source: §1 · p. 3

2. Related Work

Prior work addresses topic induction, review-property extraction, and multi-document summarization, but this paper targets noisy free-text annotations and compares documents through inferred semantic properties.

2.1 Bayesian Topic Modeling

  • Unlike unsupervised LDA, which does not link topics to external property representations, this model uses free-text annotations; unlike sLDA, it must learn from labels that are incomplete and inconsistent.This input difference motivates jointly inducing annotation structure and predicting properties from text.Source: §2.1 · p. 4

2.2 Property Assessment for Review Analysis

  • Prior review-property systems extract phrases or classify text using engineered knowledge or induced labels, whereas this approach predicts properties beyond verbatim mentions and learns relationships among keyphrases.Compared with methods requiring predefined properties and numerical annotations, it uses free-text labels and supports cross-review comparison across different surface forms.Source: §2.2 · pp. 5–6
  • Unlike integrated opinion summaries that supplement an expert article, this work produces compact summaries of user opinions without requiring an authoritative article as a template.It aims to identify underlying semantic properties rather than align user opinions to one expert review.Source: §2.2 · pp. 5–6

2.3 Multidocument Summarization

  • Where conventional multi-document summarizers identify redundancy through sentence similarity, this method abstracts reviews into properties and selects those stated by a plurality of users.The resulting property overlap supports comparisons across reviews even when their wording differs.Source: §2.3 · p. 6

3. Analysis of Free-Text Keyphrase Annotations

An analysis of restaurant-review annotations finds frequent omissions and diverse paraphrases, motivating a model that clusters keyphrases while linking them to review text.

3.1 Incompleteness

  • User annotations captured an average of 1.66 relevant properties per review, compared with 2.56 properties expressed in the text, revealing frequent omission of relevant labels.The comparison used expert judgments on 170 reviews and standardized author keyphrases for six properties.Source: §3 · pp. 7–8

3.2 Inconsistency

  • The analysis measures omission by comparing author labels with expert-supported text properties, and paraphrase variability by counting phrase forms and measuring the most common form’s coverage.The coverage estimate overstates true coverage because the analysis considers only about a tenth of keyphrase occurrences; these patterns motivate clustering phrases while relating clusters to text.Source: §3.1 · p. 8, §3.2 · p. 9

3.1 Incompleteness

  • More than 40% of property occurrences in review text lacked an explicit annotation, and precision and recall varied by property, with recall lower for good service than good food.Some keyphrases also expressed properties not mentioned in the text.Source: §3.1 · p. 8

3.2 Inconsistency

  • The dataset contains 26,801 distinct keyphrase forms across 49,310 occurrences, and the most frequent phrase for a property accounts for only about one-third of its occurrences.For good service, the top four phrases cover only three quarters of occurrences, even in the limited analyzed subset; Figure 3 plots cumulative counts for the top ten phrases.Source: §3.2 · pp. 8–9, Figure 3 · p. 9

4. Model Description

The model couples latent document topics to keyphrase clusters while retaining a background topic path for properties absent from annotations.

4. Model Description

  • Semantic topics generate document words and associate with keyphrase clusters, while allowing topics without keyphrases to accommodate incomplete annotations.At test time, annotations are hidden and the model predicts applicable keyphrases from document topics.Source: §4 · p. 11

4.1 Keyphrase Clustering

  • Keyphrases are clustered using a real-valued pairwise similarity matrix, and document topic structure also informs clustering.The matrix allows surface and distributional comparisons rather than modeling keyphrase surface forms as generated text.Source: §4.1 · pp. 11, 13, §4 · p. 11

4.2 Document Topic Modeling

  • Each document’s annotation-derived topic distribution gives equal probability to annotated topics, while a background distribution covers other topics and permits unlinked clusters.The background path lets the model account for topics not represented by keyphrase annotations.Source: §4.2 · p. 13

4.3 Generative Process

  • The generative process uses pairwise keyphrase similarities, document words, and keyphrase annotations, with enough clusters for phrase-backed and word-only topics.The model generates cluster assignments, topic and background distributions, and word-level topic assignments.Source: §4.3 · pp. 13–14
  • Similarity scores are drawn from distributions biased high for keyphrases in the same cluster and low for keyphrases in different clusters.The specified distributions are Beta(2, 1) for same-cluster pairs and Beta(1, 2) for different-cluster pairs.Source: §4.3 · p. 14

5. Parameter Estimation

The model estimates hidden parameters with conditional sampling, then predicts properties of unannotated documents from their text using the background topic distribution.

5. Parameter Estimation

  • Parameter updates use observed assignment counts: cluster counts update ψ, background and topic-specific word counts update φ_d and θ_k, and c counts update λ.The φ_d counts include only words assigned through the background model (c=0).Source: §5 · p. 15
  • Keyphrase cluster assignments are sampled conditionally on cluster priors, similarity scores, and word-topic assignments through the annotation topic distribution.For each possible cluster assignment, the model evaluates the conditional probability and samples from the normalized multinomial.Source: §5 · p. 16
  • Word-topic assignments combine word likelihoods with either annotation-derived or background topic probabilities, and an auxiliary variable selects between those distributions.The auxiliary variable is sampled using its prior and the hidden topic assignment.Source: §5 · p. 16
  • For an unannotated test document, the model estimates φ_d from text alone by fixing c=0 and predicts topics whose probabilities exceed development-set-tuned thresholds.The estimate uses the final 1000 training samples of the language models rather than a point estimate.Source: §5 · p. 16

6. Evaluation of Summarization Quality

This section evaluates PRÉCIS’s semantic-property summaries through single-document prediction, keyphrase clustering, and multi-review summarization. It assesses how joint learning affects prediction and clustering and compares multi-review summaries with annotation-based baselines when the model receives no keyphrases.

6.1.1 EXPERIMENTAL SETUP

  • Free-text evaluation treats missing keyphrase annotations as unknown and evaluates each property only on documents annotated with that property or its antonym.Across expert and free-text evaluations, automatic methods had the same relative ordering, supporting free-text labels as a proxy for relative comparisons. Tests used restaurant, cell-phone, and digital-camera reviews, with half of each domain’s documents selected for training.Source: §6.1.1 · pp. 18–19, §6.1.2 · p. 25

6.1.2 RESULTS

  • Our model significantly outperformed the three non-clustering baselines in every evaluation.The comparisons cover expert semantic annotations and author free-text annotations across the reported evaluations.Source: Table 5 · p. 22, Table 6 · p. 23, §6.1.2 · p. 24
  • Adding keyphrase clustering substantially improved baseline recall with little impact on precision, but our model still significantly outperformed model cluster in text in every evaluation.Clustering information also improved recall for keyphrase in text, including in the cameras domain, where that baseline already performed well.Source: Table 5 · p. 22, Table 6 · p. 23, §6.1.2 · p. 26
  • Cluster-based classifiers sometimes surpassed our model, while heuristic noise reduction consistently hurt classifier performance.The cluster classifier combines the model’s clustering with a more sophisticated text classifier, whereas the heuristic reduces training data.Source: Table 5 · p. 22, Table 6 · p. 23, §6.1.2 · p. 26
  • The gold-cluster model achieved F-scores comparable to our model, while baseline results gave little reason to expect substantial gains from a more gold-like clustering.This supports the sufficiency of the model-induced clustering for semantic inference.Source: Table 5 · p. 22, Table 6 · p. 23, §6.1.2 · p. 27
  • Joint inference improves clustering over keyphrase-similarity-only clustering in every domain, while independent clustering consistently degrades property prediction against the full joint model.Expert-annotated clustering does not always improve results, but joint learning of clustering and text models supports better property prediction.Source: Table 7 · p. 27, §6.1.2 · p. 27

6.2.1 DATA AND EVALUATION

  • The evaluation aggregates five user-written reviews for each of 50 restaurants, treating properties supported by at least three reviews as ground truth.Ten annotators labeled five restaurants each, using six salient properties and the prior restaurant annotation guidelines.Source: §6.2.1 · p. 27

6.2.3 RESULTS

  • Our model significantly improves multi-review F-score over all annotation-based baselines, with dramatically higher recall but reduced precision despite lacking keyphrase annotations.Exact string matching yields few predictions, while paraphrase clusters improve baseline performance; incomplete annotations still limit their recall.Source: Table 8 · p. 28, §6.2.3 · p. 29

6.2.2 AGGREGATION APPROACHES

  • Aggregation baselines require support from at least two of five reviews, using exact keyphrase matches or paraphrases from model, expert, or independent clusters.The exact-match baseline requires verbatim annotations; the model uses trained predictions without keyphrases, although baselines receive them.Source: §6.2.2 · p. 28

7. Conclusions and Future Work

The paper uses noisy novice keyphrase annotations for document-level semantic inference by linking keyphrase clusters with hidden topics in a joint hierarchical Bayesian model. It reports stronger semantic-property extraction than competitive baselines, while identifying limits in similarity modeling, property-importance prediction, and assumptions about relationships among properties.

Conclusions

  • The model jointly links keyphrase clusters and hidden text topics, using novice annotations for semantic inference and outperforming competitive baselines on single- and multi-document extraction.The authors also report improved paraphrase structure and aggregation of different keyphrase surface forms for multi-document summaries.Source: §7 · p. 29

Future Work

  • Keyphrase similarity scores are modeled as random variables, although the authors note that the scores are not independent.They suggest modeling generation of the entire similarity matrix jointly as a possible way to avoid this compromise.Source: §7 · p. 29
  • A related alternative would treat the keyphrase-similarity matrix as covariance structure, potentially constraining similar keyphrases to induce similar language models.The authors suggest a Gaussian process framework as one possible implementation.Source: §7 · p. 29
  • The model identifies expressed properties, but does not explicitly model how likely authors are to annotate each topic, leaving property-specific thresholds in post-processing.Explicitly modeling annotation likelihood is proposed as a possible extension that could avoid this post-processing step.Source: §7 · p. 30
  • The model treats semantic properties as unstructured, so it does not represent relations such as antonymy or hierarchical structure.The authors give good and bad food as an antonym pair that should not both label a restaurant review.Source: §7 · p. 30

Bibliographic Note

The article extends a prior conference publication with additional analysis, evaluation, and a multi-document review summarization system.

Bibliographic Note

  • The article adds a multi-document review summarization system, detailed analysis of free-text annotation distributions, and an expanded evaluation with another domain and additional baselines.The earlier work appeared in a conference publication.Source: Bibliographic Note · p. 30

Appendix A. Development and Test Set Statistics

The appendix describes the evaluation sets by property and distinguishes the document coverage used for gold-standard and free-text evaluations.

Appendix A. Development and Test Set Statistics

  • The gold-standard evaluation tests every property with every document, while free-text evaluation uses only documents annotated with that property or its antonym.The number of documents therefore differs by semantic property; Table 9 lists the properties and evaluation counts by domain.Source: Appendix A · p. 31, Table 9 · p. 31

Appendix B. Additional Multiple Review Summarization Results

The appendix tests multi-review aggregation with a stricter threshold of three positive reviews out of five. Under this setting, baseline F-scores drop substantially, while the model's F-score remains consistent with the earlier evaluation.

Appendix B. Additional Multiple Review Summarization Results

  • With three of five reviews required to predict a product property, baseline F-scores drop substantially, but the model's F-score remains consistent across evaluations.Table 10 compares aggregated predictions under this threshold and marks methods significantly worse than the model using approximate randomization at p ≤ 0.05.Source: Appendix B · p. 32, Table 10 · p. 32

Appendix C. Hyperparameter Settings

Appendix C reports the experiment hyperparameter settings by domain, noting that selected values were tuned on the development set and that λ0 was fixed uniformly.

Appendix C. Hyperparameter Settings

Related papers

This paper cites

Loading 1401.3457v1…