Source-linked AI summary
Learning Attitudes and Attributes from Multi-Aspect Reviews
Julian McAuley, Jure Leskovec, Dan Jurafsky
TL;DR
Online reviews usually provide a single score despite reflecting multiple product dimensions, motivating models that make aspect-specific preferences explicit. The paper introduces PALE LAGER and large multi-aspect review corpora for segmentation, summarization, and missing-rating prediction. The models scale to five million reviews while learning interpretable aspect and sentiment lexicons and achieving good performance across the considered tasks.
Problem
Single-score reviews do not identify the multiple aspects and aspect-specific sentiments underlying users’ evaluations, limiting interpretation of their preferences.
Method
PALE LAGER models explicit multi-aspect ratings, separately learns aspect and aspect-specific sentiment words, and supports several supervision settings at million-review scale.
Results
The model achieves good unsupervised performance, scales to five million reviews, and improves missing-rating prediction when aspect relationships are explicitly modeled.
Takeaways & Limitations
Interpretable aspect and sentiment lexicons support sentence-level aspect assignment, review summarization, and recovery of missing aspect ratings in real-world multi-aspect corpora.
Abstract
from arXiv · showhide
The majority of online reviews consist of plain-text feedback together with a single numeric score. However, there are multiple dimensions to products and opinions, and understanding the `aspects' that contribute to users' ratings may help us to better understand their individual preferences. For example, a user's impression of an audiobook presumably depends on aspects such as the story and the narrator, and knowing their opinions on these aspects may help us to recommend better products. In this paper, we build models for rating systems in which such dimensions are explicit, in the sense that users leave separate ratings for each aspect of a product. By introducing new corpora consisting of five million reviews, rated with between three and six aspects, we evaluate our models on three prediction tasks: First, we use our model to uncover which parts of a review discuss which of the rated aspects. Second, we use our model to summarize reviews, which for us means finding the sentences that best explain a user's rating. Finally, since aspect ratings are optional in many of the datasets we consider, we use our model to recover those ratings that are missing from a user's evaluation. Our model matches state-of-the-art approaches on existing small-scale datasets, while scaling to the real-world datasets we introduce. Moreover, our model is able to `disentangle' content and sentiment words: we automatically learn content words that are indicative of a particular aspect as well as the aspect-specific sentiment words that are indicative of a particular rating.
I. INTRODUCTION
The paper studies multi-aspect review systems to identify which text expresses each aspect, summarize ratings, and recover missing aspect ratings. PALE LAGER is designed to scale to millions of reviews while learning interpretable aspect and sentiment lexicons.
- Motivation: Multi-aspect ratings expose the dimensions behind users’ evaluations, but reviews may contain conflicting sentiments across aspects.Examples include beer reviews where taste is negative but palate is positive, requiring aspect-specific interpretation of words such as “hops” and “velvety.”
- Research tasks: The paper evaluates sentence-aspect assignment, review summarization, and recovery of missing aspect ratings.The model uses multi-aspect ratings as weak supervision for identifying which sentences discuss each rated aspect.
- Interpretability: The models separately learn aspect-descriptive words and aspect-specific sentiment words, producing interpretable lexicons without manual intervention or domain knowledge.For example, “body” can identify the feel aspect while “thick” expresses positive sentiment about it.
- Approach and data: PALE LAGER scales to corpora of five million reviews and supports unsupervised, weakly supervised, and fully supervised sentence-aspect prediction.The authors also obtain expert groundtruth labels for over ten thousand sentences.
- Main findings: The paper reports good unsupervised performance, while missing-rating recovery requires both aspect separation and explicit modeling of relationships between aspects.Separating aspects is crucial because reviews discussing multiple aspects can contain conflicting sentiments.
B. Contributions
The paper contributes large multi-aspect review datasets, scalable models, and an aspect–sentiment modeling framework. It applies these resources to segmentation, summarization, and missing-rating prediction across several review domains.
- Datasets: Approximately five million reviews with multi-aspect ratings and over ten thousand manually annotated sentences form the paper’s main dataset contribution.The reviews span BeerAdvocate, RateBeer, Amazon, and Audible, with three to six aspects per review.
- Models: The proposed models adapt high-throughput methods to train on millions of reviews and produce interpretable lexicons for each aspect and its associated sentiments.The approach separately models sentiment-neutral aspect words and aspect-specific sentiment words.
- Evaluation scope: The tasks extend prior work on sentence segmentation, review summarization, and aspect-rating prediction to complete, substantially larger datasets.Earlier approaches were limited to corpora of at most a few thousand reviews, whereas this work uses all existing reviews from its sources.
- Domains: The datasets cover beer sensory and pub attributes, Amazon toy attributes, and audiobook author and narrator ratings.Beer reviews include feel, look, smell, taste, and overall ratings; toy reviews include fun, durability, and educational value.
- Dataset properties: Aspect correlations vary across datasets, with Pubs having the lowest average pairwise correlation and some negatively correlated aspects.Price is negatively correlated with food and service in the Pubs data.
- Additional evaluation data: The method is adapted to CitySearch, where aspects and sentence-level sentiment are human-annotated rather than derived from aspect ratings.The adaptation treats four sentiment labels as ratings indexed per sentence.
A. Groundtruth labels
The paper constructs sentence-level groundtruth annotations and models aspect and sentiment words from multi-aspect review ratings. It assumes each sentence discusses a single aspect and separates aspect-specific content from rating-associated sentiment.
- 100 reviews from each dataset were manually labeled, totaling 4,324 sentences with aspect or ambiguous/irrelevant labels.
- Crowdsourced BeerAdvocate labels agreed with the authors’ labels in about 30% of cases, with κ = 0.11.The agreement was not significantly better than a random annotator.
- Two expert beer annotators achieved κ = 0.93 on the same 1,000 BeerAdvocate reviews.
- The model assumes each sentence discusses a single aspect, although word-level or paragraph-level alternatives are possible.
- The model separately learns words describing an aspect and words expressing sentiment about that aspect.The parameters θ and φ encode aspect and sentiment properties, respectively.
IV. LEARNING
The paper compares three learning schemes that use increasing amounts of sentence-label supervision. More supervision improves accuracy, but sufficient data enables good performance without supervision.
- Three learning schemes use increasing levels of supervision through sentence labels.
- More supervision leads to higher accuracy, while unsupervised learning can still obtain good performance given enough data.
A. Unsupervised Learning
Unsupervised learning jointly estimates model parameters and latent sentence-aspect assignments by maximizing corpus log-likelihood. Optimization alternates between these variables, with initialization and regularization choices supporting interpretable parameters.
- Unsupervised learning maximizes corpus log-likelihood over parameters θ, φ and latent aspect assignments t.
- Coordinate ascent alternates optimization of the parameters and latent assignments until the assignments converge.
- The model uses squared ℓ2 regularization and initializes aspect parameters to favor sentences explicitly mentioning aspect names.
- Adding a constraint to remove parameter non-identifiability has no effect on performance but produces more interpretable parameters.
- The assignment matrix is visualized as a weighted bipartite graph whose optimal cover represents sentence-to-aspect assignments.
B. Enforcing Diversity in the Predicted Output
The method enforces diversity by requiring predicted assignments to cover the review’s aspects, using weighted bipartite graph cover and linear assignment. This prevents correlated aspects from coalescing, while allowing relaxed constraints when appropriate.
- Similar aspects such as smell and taste can coalesce because correlated ratings and shared vocabulary make merging attractive to the regularizer.
- The model constrains per-review assignments so every aspect is discussed at least once, enforcing diversity through bipartite graph cover.The cover is optimized using linear assignment.
- The review graph matches sentences to aspects, with K nodes constrained to cover the K aspects and remaining nodes free to match any aspect.
- For summarization, the same objective selects K sentences most compatible with the K aspect ratings, while shorter reviews discard these constraints.
- Additional unconstrained nodes relax the coverage requirement, and datasets with easily separable aspects may discard the constraint entirely.
C. Semi-Supervised Learning
The semi-supervised variant conditions its probability model and parameter initialization on labeled data, while evaluation-oriented training directly minimizes the 0/1 loss. Because a diversity constraint couples sentence predictions, optimization requires structured-output methods rather than independent classification.
- Semi-supervised learning conditions the probability model on a fraction of groundtruth labels t′.
- The parameters θ and φ are initialized to maximize the likelihood of the observed labeled data t′.
- The fully supervised objective chooses θ̂ and φ̂ to minimize 0/1 loss on annotator-provided training data.
- Because Cohen’s kappa is monotonic in 0/1 loss, minimizing that loss maximizes the kappa statistic.
- The diversity constraint couples sentence predictions, requiring structured-output optimization instead of independent multiclass classification.
V. LEARNING TO PREDICT RATINGS FROM TEXT
The paper predicts aspect ratings from review text while addressing optional ratings, mixed sentiments, and correlations between aspects. Its model combines segmented text with explicit relationships between aspects, because separating text alone is insufficient for reliable recovery.
- 199,810 crawled reviews had overall ratings but were missing at least one optional aspect rating.The models learn from reviews with complete ratings to predict these missing values.
- A naïve unsegmented-text predictor struggles when positive and negative words jointly describe different aspects.Mixed feelings make it difficult to disentangle aspect-specific opinions.
- Segmented text alone also performs poorly because correlated aspects can make aspect-specific words predict ratings for other aspects.The word “skunky,” for example, strongly predicts low ratings across aspects despite referring specifically to smell.
- The proposed model combines segmented text with a smoothness term α that encodes how ratings co-occur across aspects.For example, α can penalize unlikely combinations such as a 1-star smell rating with a 5-star taste rating.
- The predictors are trained to minimize ℓ2 error, using multiclass SVMs for independent objectives and structured learning when α couples ratings.
- Figure 3 represents observations as grey nodes and variables as white nodes in a graphical model linking segmented text with aspect relationships.
VI. EXPERIMENTS
The experiments evaluate segmentation, summarization, and rating prediction using separate training and test splits. They compare unsupervised, semi-supervised, and fully supervised training under task-specific evaluation procedures.
- The experiments evaluate sentence-aspect segmentation, one-sentence-per-aspect summarization, and rating prediction.Segmentation and summarization use accuracy, while rating prediction uses scaled-rating ℓ2 error.
- Groundtruth data are randomly split into training and test sets, with all algorithms evaluated on held-out groundtruth labels.The rating-prediction task receives an additional split of unlabeled data.
- Unsupervised learning uses entire corpora without sentence labels, semi-supervised learning adds labeled training data, and fully supervised learning uses only labeled training data.
A. Review Segmentation
PALE LAGER improves review segmentation with increasing supervision and generally outperforms LDA, while scaling from small benchmarks to seven datasets.
- 45% average improvement comes from semi-supervised over unsupervised learning, followed by a further 17% gain from fully supervised learning.The sole exception is Audible, possibly because of overfitting.
- PALE LAGER’s unsupervised method performs poorly on non-English RateBeer data, indicating that effective unsupervised segmentation requires many reviews.Adding supervision produces similar performance across the three beer datasets.
- PALE LAGER’s semi-supervised model outperforms LDA in 5 of 7 cases and by 48% on average.Exceptions occur where reviews focus on overall evaluations rather than aspects.
- 1) Performance on CitySearch data:: On CitySearch, supervised PALE LAGER outperforms a text-only SVM by 7% while remaining competitive with sophisticated alternatives.The dataset contains 652 reviews, and the model requires only a few seconds to train.
- Sentence ranking compares unsupervised, semi-supervised, and fully supervised PALE LAGER with a fully supervised Maximum Entropy classifier.The figure reports precision-recall curves for the sentence-ranking evaluation.
1) Aspect Ranking:
For aspect ranking on BeerAdvocate, PALE LAGER approaches or surpasses Maximum Entropy performance as supervision increases.
- Aspect Ranking:: Sentence ranking retrieves the most relevant sentence for each aspect by sorting sentences according to aspect-specific probabilities.This operationalizes summarization as retrieving sentences relevant to each aspect.
- Aspect Ranking:: Mean Average Precision is 0.76 for unsupervised PALE LAGER versus 0.82 for fully supervised Maxent.The unsupervised method is close to the fully supervised classifier’s performance.
C. Rating Prediction
Rating prediction is difficult because aspect sentiments are correlated, but PALE LAGER improves performance by explicitly modeling relationships between aspects.
- Segmented text alone is 32% worse than unsegmented text for rating prediction, even with accurate aspect labels.Conflicting sentiments and correlated aspect ratings explain why segmentation alone is insufficient.
- 23% lower error than the SVM baseline results from combining the pairwise rating model with unsegmented text.Adding segmented text reduces error by a further 22%.
- Explicitly modeling relationships between aspects largely addresses the correlation problem in missing-rating prediction.The pairwise rating model is designed to account for correlations that segmented-text prediction misses.
- Supervision has little impact on rating-prediction performance, despite substantially affecting segmentation performance.Good rating prediction can occur even when aspect labels are inaccurate because unsupervised segmentation learns words correlated with ratings.
- D. Qualitative Analysis: The learned BeerAdvocate lexicons separate aspect words, such as nouns, from sentiment words, such as adjectives.Examples include carbonation, head, aroma, and flavor as aspect-indicative words.