Source-linked AI summary
Yelp Dataset Challenge: Review Rating Prediction
Nabiha Asghar
TL;DR
Review Rating Prediction aims to infer a user's star rating from the text of that user's review, a difficult task because users can give the same rating for different reasons. The paper evaluates sixteen combinations of semantic text features and supervised classifiers on Yelp reviews. Logistic regression with the top 10,000 Unigrams & Bigrams achieves the best reported accuracy of 64%, while the study identifies linear models and omission of sentiment analysis as scope boundaries.
Problem
Review Rating Prediction asks whether a review's free-form text can predict its numeric star rating, despite different users giving the same rating for different reasons.
Method
The paper treats rating prediction as multi-class classification and combines four semantic feature methods with four supervised learning algorithms to evaluate sixteen models on Yelp data.
Results
64% accuracy is achieved by Logistic Regression using the top 10,000 Unigrams & Bigrams, the best reported result among the evaluated models.
Takeaways & Limitations
The selected system can generate star ratings on review websites where users write free-form reviews without specifying a rating.
Takeaways & Limitations
All prediction models are linear, and the paper does not incorporate sentiment analysis into its semantic text analysis.
Abstract
from arXiv · showhide
Review websites, such as TripAdvisor and Yelp, allow users to post online reviews for various businesses, products and services, and have been recently shown to have a significant influence on consumer shopping behaviour. An online review typically consists of free-form text and a star rating out of 5. The problem of predicting a user's star rating for a product, given the user's text review for that product, is called Review Rating Prediction and has lately become a popular, albeit hard, problem in machine learning. In this paper, we treat Review Rating Prediction as a multi-class classification problem, and build sixteen different prediction models by combining four feature extraction methods, (i) unigrams, (ii) bigrams, (iii) trigrams and (iv) Latent Semantic Indexing, with four machine learning algorithms, (i) logistic regression, (ii) Naive Bayes classification, (iii) perceptrons, and (iv) linear Support Vector Classification. We analyse the performance of each of these sixteen models to come up with the best model for predicting the ratings from reviews. We use the dataset provided by Yelp for training and testing the models.
1. Introduction
Review Rating Prediction asks whether a user's free-form review text can explain and predict the numeric star rating for a specific business. The paper frames this difficult task as multi-class classification and evaluates sixteen combinations of text features and supervised learning algorithms on Yelp data.
- Online reviews combine free-form text with numeric star ratings and influence consumer purchase decisions, product sales, and business revenues.
- Readers often rely on star ratings because businesses may receive hundreds of reviews, but the relationship between review text and ratings is not obvious.
- Review Rating Prediction learns a mapping from a review's word vector to its numeric rating for a specific product or business.
- The central challenge is extracting useful product features from text and quantifying their relative importance to the rating despite users giving identical ratings for different reasons.
- The paper builds sixteen models by combining unigrams, bigrams, trigrams, and Latent Semantic Indexing with four supervised learning algorithms, then evaluates them on Yelp data.
2. Related Work
Prior review-rating research commonly extracts sentiment or reviewer and product features, using datasets and targets that differ from this paper's restaurant review-rating task. This paper explicitly focuses on semantic analysis rather than sentiment analysis.
- Recent review-rating studies often use sentiment analysis, including opinion extraction, sentiment scoring, and sentiment-class-specific word frequencies.
- Related methods combine text-derived features with collaborative filtering or domain-dependent models and evaluate them on Amazon or IMDb reviews.
- Business rating prediction on Yelp differs from review rating prediction because it predicts a restaurant's average star rating rather than an individual review's rating.
- Other work adds reviewer and product or business features, whereas this paper concerns only semantic analysis of review text and excludes sentiment analysis.
3. Data Description
The study uses Yelp Dataset Challenge 2014 data covering multiple object types, but extracts the necessary business and review information for restaurant-focused review-rating prediction. Restaurants account for 68.3% of the reviews, motivating this scope restriction.
- The Yelp dataset contains 42,153 businesses and 1,125,458 text reviews from Phoenix, Las Vegas, Madison, Waterloo, and Edinburgh.
- The dataset has separate JSON-lines files for businesses, reviews, users, check-ins, and tips, but the study uses only business.json and review.json.
- Businesses span categories such as restaurants, shopping, and hotels, whose review vocabularies may differ substantially.
- Restaurants comprise almost 34% of businesses and 68.3% of reviews, so the paper restricts Review Rating Prediction to restaurants.
4. Experimental Setup
The experimental setup constructs sixteen prediction systems by pairing four feature-extraction methods with four supervised learning algorithms. The methods include preprocessing, semantic feature construction, learning, and evaluation using two performance metrics.
- The study combines four feature extraction methods with four supervised learning algorithms to build sixteen prediction models.The setup also describes preprocessing and two performance evaluation metrics.
4.1. Preprocessing
The paper separates restaurant data from Yelp’s business and review files, then preprocesses free-form reviews to extract meaningful content.
- Restaurant businesses and their reviews are first separated from Yelp’s business.json and review.json files.
- Reviews are preprocessed because free-form text may contain excessive capitalization, punctuation, slang, and frequent stop words.
4.2. Feature Extraction
The paper builds review feature vectors using lexical n-grams and Latent Semantic Indexing, with TF-IDF weighting for word- and phrase-based representations.
- Four semantic-analysis methods extract useful features and construct a feature vector for each review.
- Unigrams: Unigrams represent unique words, and TF-IDF weighting reduces the influence of common words while emphasizing rarer words.
- Unigrams & Bigrams: Bigrams extend unigrams with consecutive word pairs to capture phrase effects such as modifiers and negation.
- Trigrams: Trigrams add consecutive three-word phrases, but their rarity across reviews means results are not expected to differ greatly from the unigrams-plus-bigrams model.
- Latent Semantic Indexing (LSI): LSI constructs a word-review matrix from unigrams, applies SVD, and can use the most important topics as a reduced feature matrix.
4.3. Supervised Learning
The paper trains four supervised classifiers for multi-class review-rating prediction, including probabilistic, linear, and margin-based approaches.
- Four supervised learning algorithms are used to train the prediction models.
- Logistic Regression: Logistic regression models P(s|r) and predicts the star-rating class with the highest probability.
- Naive Bayes: Multinomial Naive Bayes models review-rating probabilities under conditional feature independence and is suited to count-based text features.
- Perceptrons: The perceptron is a linear classifier that repeatedly updates on misclassified training examples.
- Linear SVM: Linear SVMs optimize the decision-boundary margin, allow misclassifications, and select C by internal 3-fold cross-validation.
4.4. Performance Metrics & Implementation Details
The evaluation uses an 80/20 train-test split, 3-fold cross-validation on training data, RMSE and accuracy, and a specified Python computing environment.
- Evaluation: The dataset is divided into 80% training data and 20% testing data.
- Evaluation: Each of the sixteen systems is evaluated with 3-fold cross-validation using RMSE and accuracy on training and validation folds.
- Implementation: Implementation uses Python with numpy, scipy, and scikit-learn on a four-core Intel Core i5 system with 8 GB RAM.
5. Results and Analysis
Across feature representations, logistic regression generally performs best, with unigrams and bigrams producing the strongest validation results; LSI shows unresolved patterns beyond 200 features, while test performance declines.
- Unigrams: Validation RMSE and accuracy level off at about 10,000 unigram features, except Naive Bayes, whose RMSE rises after reaching a minimum there.Training-fold performance worsens as features increase, while validation performance stabilizes.
- Unigrams: Logistic regression leads unigram validation with RMSE 0.85 and accuracy 58%, narrowly ahead of Linear SVC at 0.87 and 57%.Perceptrons perform worst at RMSE 1.25 and accuracy 43%; Naive Bayes reaches 0.96 and 52%.
- Unigrams & Bigrams: Unigrams and bigrams improve RMSE and accuracy for every classifier, with logistic regression reaching 0.78 RMSE and 64% accuracy.Bigrams occur frequently enough to capture information that unigrams miss; logistic regression again outperforms Linear SVC, which reaches 0.81 and 63%.
- Unigrams, Bigrams & Trigrams: Adding trigrams does not help because repeated three-word phrases are rare and TF-IDF therefore assigns them mostly rare feature weights.The trigram results are almost exactly the same as those for unigrams and bigrams, with logistic regression achieving 0.78 RMSE and 64% accuracy.
- Latent Semantic Indexing (LSI): LSI performance suggests that the top 200 topics are most important, but accuracy trends and unexplained spikes indicate that more than 200 features should be tested.The authors leave this extension as future work because of time constraints.
- Best Model and Test Set: The best validation model is logistic regression with the top 10,000 unigrams and bigrams, achieving 64% accuracy versus 63% for Linear SVC.On the test set, Linear SVC scores RMSE 1.05 and accuracy 56%, while logistic regression scores 0.92 and 54%, possibly indicating overfitting.
6. Conclusions & Future Work
The paper identifies logistic regression on the top 10,000 Unigrams & Bigrams features as the strongest of sixteen Yelp review-rating systems. It outlines future work spanning richer feature extraction, nonlinear and ordinal models, broader evaluation, regularization, and expanded validation or domains.
- Conclusions: Logistic Regression on the top 10,000 features obtained from Unigrams & Bigrams exhibits better predictive powers than the other fifteen systems.
- Future Work: The tested methods are extensive but not exhaustive, leaving multiple avenues for improvement and future work.
- Future Work: Feature engineering could use POS tagging, spell-checking, selected word pairs, and fewer carefully chosen features to address efficiency and memory bottlenecks.The implementation handled up to 32 million features, and plots took 36 to 48 hours.
- Future Work: Future LSI experiments could examine more than 200 features and apply SVD to additional n-grams or selected text constructs.
- Future Work: Ordinal logistic regression could account for ordered ratings, while nonlinear models or expanded feature spaces could provide alternatives to the current linear models.