Source-linked AI summary
Addressing Complex and Subjective Product-Related Queries with Customer Reviews
Julian McAuley, Alex Yang
TL;DR
Product queries often concern subjective experiences or details that knowledge bases and product specifications cannot answer, while relevant evidence is buried in large review collections. Moqa treats reviews as experts, jointly learning relevance and voting functions from answered questions to surface useful opinions. It addresses binary and open-ended queries, with reviews outperforming product specifications and learned relevance outperforming hand-crafted measures in the reported evaluations.
Problem
Users ask complex, subjective, and niche product questions for which factual knowledge bases may be inadequate, while relevant review evidence is difficult to find.
Method
Moqa formulates review retrieval as a mixture-of-experts problem, jointly learning review relevance and answer-voting functions from previously answered questions.
Results
Moqa addressed both binary and open-ended questions; reviews outperformed product specifications, and human evaluators preferred learned relevance over hand-crafted measures.
Takeaways & Limitations
Learned relevance can help users navigate multiple subjective viewpoints and personal experiences rather than requiring a conclusive fact-based answer.
Takeaways & Limitations
Training currently uses the top-voted answer as authoritative, leaving multiple or inconsistent subjective answers for future handling.
Abstract
from arXiv · showhide
Online reviews are often our first port of call when considering products and purchases online. When evaluating a potential purchase, we may have a specific query in mind, e.g. `will this baby seat fit in the overhead compartment of a 747?' or `will I like this album if I liked Taylor Swift's 1989?'. To answer such questions we must either wade through huge volumes of consumer reviews hoping to find one that is relevant, or otherwise pose our question directly to the community via a Q/A system. In this paper we hope to fuse these two paradigms: given a large volume of previously answered queries about products, we hope to automatically learn whether a review of a product is relevant to a given query. We formulate this as a machine learning problem using a mixture-of-experts-type framework---here each review is an `expert' that gets to vote on the response to a particular query; simultaneously we learn a relevance function such that `relevant' reviews are those that vote correctly. At test time this learned relevance function allows us to surface reviews that are relevant to new queries on-demand. We evaluate our system, Moqa, on a novel corpus of 1.4 million questions (and answers) and 13 million reviews. We show quantitatively that it is effective at addressing both binary and open-ended queries, and qualitatively that it surfaces reviews that human evaluators consider to be relevant.
1 Introduction
Moqa learns which product reviews are relevant to subjective, complex, and long-tail queries, using answered questions to train review relevance and voting functions. It surfaces relevant opinions for binary answers and open-ended questions, helping users form conclusions themselves.
- Product reviews provide objective properties, subjective assessments, and distinctive use or failure experiences beyond simple good-or-bad judgments.
- Users need help navigating massive review collections to find opinions relevant to specific product-related queries.
- Moqa learns review relevance from previously answered community questions, while reviews vote on responses in proportion to their learned relevance.The voting function is needed during training, although test-time retrieval requires only the relevance function.
- Subjective, long-tail, and linguistically complex queries make fact repositories and word-level similarity insufficient for retrieving useful reviews.Relevant reviews may use different vocabulary from the query, requiring learned handling of subtle differences such as synonyms.
- The mixture-of-experts framework treats individual reviews as experts and jointly learns their relevance and prediction functions from training data.
- Moqa is evaluated on 1.4 million answered questions and 13 million reviews for binary and open-ended queries, with quantitative and human-evaluation results.The authors compare against state-of-the-art relevance-ranking methods and assess whether humans agree with the learned relevance notion.
2 Related Work
Moqa connects opinion mining, relevance ranking, and question answering by using consumer reviews to address product queries. Unlike systems that retrieve objective facts or produce query-independent summaries, it learns query-specific relevance across heterogeneous text sources.
- The work lies at the interface of opinion mining and question answering, using consumer reviews to answer general product queries.
- Document summarization: Review summarization commonly extracts salient evaluative text, but most approaches are not focused on a specific query.
- Relevance ranking: Query-focused relevance ranking determines whether documents are relevant in the context of a particular query, with relevance defined in several ways.
- Opinion mining: Related opinion-mining work identifies product facets, helpful reviews, experts, or summaries, while Moqa learns relevance for answering particular queries.
- Q/A systems: Related question-answering systems select phrases, experts, answers, or previously answered questions, whereas Moqa selects reviews as evidence for subjective viewpoints.
- Key differences: Unlike knowledge-base and fact-retrieval systems, Moqa uses Q/A data to learn useful relevance functions for navigating multiple personal experiences.
3 Model preliminaries
The preliminaries introduce standard relevance-ranking measures, bilinear models for cross-vocabulary compatibility, and mixtures of experts for combining review predictions. These components support learning relevance and voting jointly for heterogeneous questions, reviews, and answers.
- Standard measures for relevance ranking: Cosine similarity and BM25 provide standard query-document relevance measures for comparing questions with reviews.BM25 addresses common-word dominance through term-frequency and inverse-document-frequency weighting.
- Bilinear models: Bilinear models learn mappings between query and document feature spaces to capture relationships such as synonyms across different vocabularies.A low-rank approximation M ∼ AB^T makes the high-dimensional relationship matrix practical while projecting related terms into a shared space.
- Mixtures of experts: Mixtures of experts combine classifier outputs using confidence weights, treating individual reviews as experts that support or oppose query responses.The framework jointly learns relevance and classification parameters so experts can focus on instances they classify well.
- Mixtures of experts: Each expert’s confidence is derived from a relevance score using softmax, while binary predictions are obtained with a logistic function.The relevance and voting functions are parameterized and trained to maximize the likelihood of available labels.
4 MOQA
Moqa treats reviews as experts that both vote on product queries and receive learned relevance weights. It supports binary and open-ended questions by learning from answered queries while surfacing reviews at test time without requiring candidate answers.
- Model: Moqa simultaneously learns review relevance and query-answer predictions within a mixture-of-experts framework.Reviews are treated as experts whose predictions should be correct when their relevance is high.
- Relevance ranking: The relevance function combines pairwise similarity features with a bilinear model over query and review features.It uses similarity measures such as BM25+ and ROUGE-L, plus bag-of-words features and a low-rank approximation.
- Binary questions: For binary queries, reviews cast positive or negative votes, while the relevance function determines how much weight each vote receives.The model learns relevance and prediction parameters jointly to maximize the likelihood of correct training answers.
- Open-ended questions: For open-ended questions, Moqa ranks candidate answers by favoring the true answer over randomly sampled non-answers using pairwise ranking.The objective approximately maximizes average AUC, and answers are used only during training to learn a relevance function for later review retrieval.
- Open-ended questions: At test time, Moqa can surface relevant reviews for new queries without requiring candidate answers to be available.Candidate answers serve as training supervision rather than live-system inputs.
5 Experiments
Moqa is evaluated on binary-query resolution, open-ended answer selection, and human judgments of review relevance using Amazon reviews and product questions.
- Evaluation design: Moqa is evaluated on binary queries, open-ended queries, and qualitative human assessments of review relevance.The evaluation covers resolving yes/no questions, selecting correct answers among alternatives, and identifying reviews humans judge relevant.
5.1 Data
The dataset combines Amazon product reviews with community questions and top-voted answers across a large collection of products.
- Dataset: 1.4 million questions and answers from 191 thousand products are paired with over 13 million customer reviews.The collection also includes product descriptions for comparing description text with review text.
5.2 Labeling yes/no answers
Binary-question training requires identifying questions and answers with reliable yes/no labels. Manual analysis and high-precision classifiers provide these labels, yielding 309,419 high-confidence training instances.
- Labeling procedure: The manually labeled sample was used to evaluate automatic labeling methods rather than train Moqa directly.The labeling task determines whether questions and answers are binary and whether answers are positive or negative.
- Manual labeling: 56.1% of sampled questions were binary, and 76.5% of those had conclusive binary answers.Among questions with yes/no answers, 62.4% had positive answers.
- Question classification: 97% precision at 82% recall was achieved for identifying yes/no questions with grammatical rules.The procedure favors precision over recall because only high-confidence labels are retained.
- Answer classification: 98% held-out accuracy was obtained for yes/no answer classification after retaining the 50% most confident instances.The classifier used bag-of-unigrams, the first word, and confidence-based filtering.
- Final training set: 309,419 questions were labeled with high confidence for training the binary version of Moqa.A question was retained only when it passed both the binary-question and high-confidence answer tests.
5.3 Baselines
Moqa is compared with retrieval and learned relevance baselines to assess ranking measures, training, data source, and bilinear relevance modeling. Reviews are split into sentences for more targeted surfacing and slightly better performance than complete reviews.
- Learning versus non-learning: Maximum-likelihood training tunes the parameters of these relevance measures, producing learned variants such as c-L, o-L, and r-L.The comparison distinguishes off-the-shelf measures from versions optimized using training data.
- Data sources: Mdqa replaces reviews with product descriptions to test whether reviews provide a better data source than product specifications.The model structure remains the same while the textual source changes.
- Relevance measures: The baselines test off-the-shelf cosine, BM25+, and ROUGE-L relevance measures.Cosine similarity, Okapi-BM25+, and ROUGE-L represent alternative ranking signals.
- Model comparison: The cro-L baseline combines cosine similarity, ROUGE-L, and BM25+ with learned weights, while Moqa additionally evaluates bilinear relevance modeling.This comparison isolates the value of bilinear models because Moqa already includes the standard similarity features.
- Result granularity: All methods rank sentences rather than complete reviews because sentence-level results are more targeted to specific product details and perform slightly but consistently better.Sentence splitting is also more convenient for surfacing results through an interface.
5.4 Quantitative evaluation
Moqa is evaluated on binary and open-ended product queries, with confidence-based classification and answer-ranking metrics. It outperforms the strongest baselines, and review text substantially outperforms product descriptions.
- Yes/no questions: Binary evaluation is difficult because some queries are not addressed by the available reviews.Probabilistic confidence scores are therefore used to assess whether models identify queries they can correctly address.
- Yes/no questions: Accuracy@50% shows Moqa substantially outperforming learned baselines, especially on larger datasets with more data for learning the bilinear map.Only learned methods are compared because non-learning approaches cannot determine suitable binary decision parameters.
- Yes/no questions: Around 90% accuracy is achieved by Moqa on the highest-confidence binary queries, exceeding every baseline across the confidence spectrum.Performance declines for low-confidence queries, but Moqa remains more accurate than alternatives throughout.
- Open-ended questions: Moqa strictly dominates all open-ended-query baselines, reducing error over the strongest baseline by 50.6% on average.The strongest baseline combines learned ROUGE and BM25+ scores; adding weighted cosine similarity does not further improve it.
- Reviews versus product descriptions: Moqa with review text reduces error by 37.5% relative to Mdqa using product descriptions.Mdqa performs on par with the strongest baselines but remains substantially below Moqa with reviews for both query types.
- Reviews versus product descriptions: Reviews are effective partly because they capture subjective experiences and are roughly 100 times more abundant than product descriptions.Their volume and diversity provide broader coverage for answering questions.
5.5 Qualitative evaluation
The qualitative evaluation tests whether Moqa’s learned relevance function aligns with human judgments of useful opinions. In a Mechanical Turk study, users preferred Moqa’s surfaced opinions over the strongest baseline in most cases.
- User study: The study compares Moqa’s top-ranked result with the top result from the tuned BM25+ and ROUGE-L baseline, plus a random result and an option to select no result.Cases with overlapping results were discarded.
- User study: Moqa was preferred in 73.1% of instances across six datasets in the user study.Preference ranged from 66.2% on Sports and Outdoors to 77.6% on Baby.
- User study: Moqa’s preference advantage was similar for objective and subjective queries, at 74.1% and 62.9%, respectively.The evaluation used 100 queries from each of five largest datasets plus one smaller Baby dataset.
- Examples: Held-out examples show that Moqa must learn relevance across varied products, queries, opinions, and heterogeneous language.Examples include semantic relations such as “girl” and “tall” matching “daughter” and “medium.”
- Efficiency: At test time, Moqa computes relevance in O(K + |q| + |r|), enabling responses in a few milliseconds for products with thousands of reviews.All reviews can be projected through B in advance.
6 Discussion and Future Work
The discussion examines surprising evaluation patterns and proposes extending Moqa with richer review-site information, compatibility modeling, and support for inconsistent answers.
- Discussion: Open-ended queries performed better than binary queries against random classification and the strongest baseline, possibly because more training data was available.The authors attribute the relative baseline improvement to the bilinear model’s greater parameterization and the larger open-ended training set.
- Discussion: User-study performance was roughly equal on subjective and objective queries, while many apparently objective questions received contradictory answers reflecting different user experiences.Only around 20% of queries were labeled subjective, and the authors note that questions such as duration queries can elicit varied responses.
- Future work: Additional rating information could improve performance on certain evaluative queries, although sentiment polarity already captures some rating-related information.User identities could also support relevance models by representing aligned opinions or identifying topical experts.
- Future work: Compatibility modeling is proposed for electronics queries because the large number of product combinations makes user reviews a potentially suitable data source.The current system can already address compatibility queries to some extent, but the authors identify a dedicated compatibility model as desirable.
- Future work: Handling multiple and inconsistent answers could identify subjective or contentious questions and produce relevance rankings spanning subjective viewpoints.The current training procedure selects the top-voted answer as authoritative.
7 Conclusion
Moqa learns to surface relevant consumer opinions for product-related queries by treating reviews as voting experts and learning relevance from answered questions. Evaluation found reviews effective across binary and open-ended questions, with learned relevance preferred to hand-crafted measures.
- Conclusion: Moqa automatically responds to product-related queries by surfacing relevant consumer opinions learned from previously answered questions.The system casts relevance and answer prediction as a mixture-of-experts problem in which opinions vote in proportion to relevance.
- Conclusion: Reviews outperformed product specifications as a source for answering product-related queries, demonstrating the value of personal experiences.The evaluation also identified heterogeneity among questions, reviews, and answers as requiring explicit handling.
- Conclusion: Moqa quantitatively addressed both binary and open-ended questions, while human evaluators qualitatively preferred its learned relevance over hand-crafted relevance measures.The learned model accounts for linguistic differences among text sources through a flexible bilinear formulation.