Ranking Online Consumer Reviews

1. Introduction

• 80% of customers read product reviews before purchasing, while 90% read ten or fewer, according to the cited BrightLocal survey.The authors use these figures to motivate why review-list ordering matters when products have many reviews.Source: §1 · p. 2
• Vote-based ordering can reinforce a Matthew effect: visible reviews attract votes while overlooked reviews remain near the bottom.The authors argue that content-based sorting is needed to mitigate this effect and give reviews fair visibility.Source: §1 · p. 3
• The proposed hybrid model classifies reviews as high or low quality, then trains its helpfulness regressor using only high-quality reviews.The paper argues this better fits the goal of ranking high-quality reviews and avoids needless list changes for low-quality reviews.Source: §1 · p. 4
• Product-description attributes and customer question-answer topics offer additional signals for matching review content to buyer interests.The authors propose matching review sentences to product features using similarity and compare review text with customer questions and answers.Source: §1 · pp. 4–6
• The study asks whether excluding low-quality reviews improves regression, and whether product-description and question-answer features affect helpfulness prediction.It also identifies class imbalance between high- and low-quality reviews as a problem addressed after labeling.Source: §1 · p. 6

2.1 Automated approach for helpfulness prediction

• Automated studies used statistical and supervised models, relating helpfulness to sentiment, review depth, ratings, reviewer attributes, and textual features.The review also describes neural-network and regression approaches, including Lee and Choeh (2014) and Ngo-Ye and Sinha (2014).Source: §2.1 · pp. 7–10

2.2 Manual approach for helpfulness prediction

• Manual studies used surveys and laboratory experiments; Mackiewicz and Yeats (2014) found informativeness the most important component of review quality.Other cited findings include higher perceived helpfulness for concrete than abstract reviews.Source: §2.2 · p. 11

Literature gap and contribution

• Earlier work generally used classification to separate review quality or regression to predict the best reviews, whereas this paper combines both stages.The authors first classify reviews, then send only high-quality reviews to a regressor.Source: §2.2 · p. 11
• The authors identify customer question-answer data as an unexamined helpfulness feature and report that product-description and question-answer features affect review ranking.Their review also notes earlier determinants such as product type, review extremity, review text, reviewer reputation, and sales.Source: §2.2 · p. 12

3. Research Methodology

• The methodology’s five components are data collection, cleaning, feature extraction, classification, and rank prediction.The workflow is presented in Figure 2.Source: §3 · p. 12, Figure 2 · p. 14

3.1. Data collection

• The study collected 29,215 Amazon reviews and 12,686 Snapdeal reviews for five products from October to December 2016.Collected data included product descriptions, question-answer data, and review text.Source: §3.1 · p. 12

3.2. Data cleaning

• Data cleaning removes Unicode characters and product images, caps exceptionally high vote counts, and leaves 27,190 Amazon and 10,255 Snapdeal reviews.The cap is three times the non-zero average helpfulness votes.Source: §3.2 · p. 14

3.3. Feature extraction

• The 17 features comprise 15 review-data features plus Desc_sim and QA_sim, which measure similarity to product descriptions and customer questions.Cosine similarity is calculated from bag-of-words document vectors.Source: §3.3 · p. 15

3.4. Classification

• Reviews are labeled high- or low-quality using a product-specific threshold set to the average helpfulness.The four classification cases compare review-only features against adding product-description data, question-answer data, or both; SVM, naïve Bayes, and random forest are tested.Source: §3.4 · pp. 19–20

3.5. Rank prediction

• Rank prediction compares linear regression and gradient boosting, with and without classification, using a 3:1 training-to-testing split.In the classifier-based route, only high-quality reviews go to the regressor; without classification, all reviews are ranked.Source: §3.5 · p. 20

4. Result

• Classifier evaluation used precision, recall, F1-score, and ROC curves, while regression performance was measured by mean square error.Source: §4 · pp. 20–21

4.1. Rank prediction with classifier

• Class 1 F1-score reached 0.93 for Amazon Case 4 and 0.85 for Snapdeal Case 4, with both description and question-answer features included.For Amazon, the individual-feature cases scored 0.91 and 0.88; for Snapdeal, they scored 0.78 and 0.74, versus 0.65 for baseline.Source: §4.1 · p. 23, Table 4 · p. 23, Figure 6 · p. 25
• The gradient-boosting regressor, trained on Case 4 high-quality reviews, obtained 0.267 MSE for Amazon and 0.623 MSE for Snapdeal.The study reports that linear regression performed quite poorly compared with gradient boosting.Source: §4.1 · p. 26
• The predicted and actual top-ten lists averaged 5.66 matching reviews for Amazon and 6.12 for Snapdeal.The matching score was calculated from the intersection of the predicted and actual top-k lists, with k set to 10.Source: §4.1 · pp. 26–27

4.2. Rank prediction without classifier

• Without the classifier, MSE rose from 0.267 to 2.545 for Amazon and from 0.623 to 3.434 for Snapdeal; average top-ten matching fell to 5.23 and 4.32, respectively.Source: §4.1 · p. 26, §4.2 · p. 27

4.3. Testing

• 80% of the reader annotations favored the predicted top ten, which averaged 6.87 matches with the actual top ten on the additional Amazon test set.The test used 1,000 additional Amazon reviews, and the predicted list was judged more effective in 80% of cases.Source: §4.3 · pp. 27–28

5. Discussion

• The system classifies reviews into high- and low-quality classes, trains the regressor only on high-quality reviews, then ranks those reviews by predicted helpfulness and appends the low-quality reviews.The study used SMOTE to balance the classes and reported its best classification results with a random-forest classifier.Source: §5 · p. 28
• 0.93 F1-score (Amazon) and 0.85 F1-score (Snapdeal) were the best Case 4 classifier results, versus baselines of 0.86 and 0.65, respectively.Case 4 combined review text, product-description data, and customer question-answer data.Source: Table 6 · p. 29, §5 · p. 30
• MSE was 0.267 for Amazon and 0.623 for Snapdeal when the regressor used only high-quality reviews, versus 2.545 and 3.434 when it used both classes.The authors also report that product-description and question-answer topics were associated with high votes in positive reviews, while the product-description effect was neutral for negative reviews.Source: §5 · p. 31
• 5–6 reviews matched between the systems’ top tens, while the proposed ranking added 4–5 newer, low-vote reviews; human evaluators rated its top ten better.The comparison used k = 10.Source: §5 · p. 31

5.1 Implications for theory

• The paper’s theoretical contribution includes a review-placement mechanism and two features—product-description and customer question-answer data—reported to have a positive association with helpfulness.The authors note that the product-description effect on negative reviews was neutral and that further analysis is needed to explain it.Source: §5.1 · p. 32

5.2 Implications for practice:

• The authors suggest that website developers could include product-description and question-answer topics on review pages and run the ranking model in the background without changing page layout.They also describe a separate linked list for top-quality reviews as one implementation option.Source: §5.2 · pp. 32–33

• Product-description data was important for positive reviews but neutral for critical reviews, while question-answer data showed a positive association with helpfulness.Source: §6 · p. 33
• The system classifies reviews by quality, predicts helpfulness for high-quality reviews, and uses those scores to place reviews in the top 10 regardless of posting date.Low-quality reviews are appended to the end of the review queue.Source: §6 · p. 33
• The findings are limited to Snapdeal and Amazon, one data-collection window, and five product categories, so generalization to other contexts requires validation.Source: §6 · pp. 33–34
• Future work could test other classification and regression techniques, deep learning, additional product or review features, and greater weight for recent high-quality reviews.Source: §6 · p. 34