PaperlayerView exact revision on arXiv

Source-linked AI summary

Mining User Opinions in Mobile App Reviews: A Keyword-based Approach

Phong Minh Vu, Tam The Nguyen, Hung Viet Pham, Tung Thanh Nguyen

arXiv:1505.04657v2cs.IRcs.CL

TL;DR

Mobile-app reviews are valuable but difficult to mine because they are voluminous and noisy. MARK addresses this with a semi-automated keyword-based framework for extracting, organizing, searching, and monitoring review opinions. Its case studies report 83% clustering accuracy, 89.7% expansion accuracy, 90% relevance among the top 50 search results, and correct detection of a real Facebook Messenger problem.

  • Problem

    Manual analysis of mobile-app reviews is challenging because popular apps generate thousands of noisy reviews daily and more than 60% reportedly contain no useful opinions.

  • Method

    MARK is a semi-automated keyword-based framework that extracts and ranks keywords, groups or expands them semantically, retrieves relevant reviews, and visualizes temporal abnormalities.

  • Results

    Case studies report 83% accuracy for keyword grouping, 89.7% for expansion, 90% of top 50 search results satisfying a battery-consumption query, and correct detection of a real Facebook Messenger problem.

  • Takeaways & Limitations

    MARK can help developers map app concerns to users’ keyword expressions while reducing effort in discovering and understanding user opinions.

Abstract

from arXiv · show

User reviews of mobile apps often contain complaints or suggestions which are valuable for app developers to improve user experience and satisfaction. However, due to the large volume and noisy-nature of those reviews, manually analyzing them for useful opinions is inherently challenging. To address this problem, we propose MARK, a keyword-based framework for semi-automated review analysis. MARK allows an analyst describing his interests in one or some mobile apps by a set of keywords. It then finds and lists the reviews most relevant to those keywords for further analysis. It can also draw the trends over time of those keywords and detect their sudden changes, which might indicate the occurrences of serious issues. To help analysts describe their interests more effectively, MARK can automatically extract keywords from raw reviews and rank them by their associations with negative reviews. In addition, based on a vector-based semantic representation of keywords, MARK can divide a large set of keywords into more cohesive subsets, or suggest keywords similar to the selected ones.

I. INTRODUCTION

MARK frames mobile-review analysis as a keyword-based information-retrieval task that helps analysts find relevant opinions despite review volume and noise. It also supports keyword discovery, semantic organization, review search, and temporal abnormality detection.

  • Motivation: Millions of mobile-app reviews are difficult to analyze manually because popular apps receive thousands daily, reviews are noisy, and over 60% reportedly lack useful opinions.Noise includes typos, acronyms, abbreviations, and emoji icons.
  • MARK framework: MARK lets analysts specify app interests with keywords and retrieves reviews whose tf.idf vectors have high cosine similarity to the keyword set.The framework treats review mining as information retrieval and uses the Vector Space Model.
  • Keyword recommendation: MARK extracts and ranks keywords associated with negative reviews, then clusters related keywords or expands selected keywords using semantic similarity.Word2Vec representations and K-means support clustering and expansion.
  • Trend analysis: MARK analyzes keyword occurrences over time with simple moving averages and flags sudden changes when deviations exceed the standard deviation of SMA values.The method is intended to identify unusual patterns that may accompany serious app issues.
  • Processing pipeline: MARK’s processing pipeline crawls and tokenizes app-store reviews, normalizes common errors and abbreviations, filters non-English reviews, and supports downstream analysis.The supplied architecture caption identifies this as MARK’s system architecture and processing pipeline.

II. KEYWORD EXTRACTION

MARK’s keyword-extraction stage addresses the spelling variation and informal language common in mobile-app reviews. It uses custom normalization resources informed by large-scale review analysis and contextual correction.

  • Review normalization: The extraction stage is designed to address misspellings, typos, acronyms, abbreviations, slang, and non-English text before extracting keywords.The paper presents this as a customized preprocessing and keyword-extraction procedure.
  • Review normalization: Mobile reviews frequently contain typos, acronyms, and abbreviations because users type on small devices without physical keyboards.These forms complicate reliable keyword extraction.
  • Review normalization: Analysis of 300,000 Google Play reviews found nearly 5,500 frequent out-of-dictionary words, including misspellings, abbreviations, and app names.The words occurred at least 20 times and were compared against an English dictionary of more than 150,000 common words.
  • Review normalization: MARK uses a custom dictionary to map frequent out-of-dictionary words to plausible corrections, using review context when the correction is ambiguous.The dictionary was developed through manual analysis by four researchers.

2) Non-English reviews:

MARK filters non-English reviews and normalizes English review text before keyword analysis. Its customized stemming preserves word meaning while correcting inflections and residual over-stemming.

  • Non-English review filtering: The preprocessing pipeline focuses on English reviews, excluding non-English reviews as noise for analysis.
  • Non-English review filtering: MARK develops a custom filter because reviews may contain English-looking words in non-English text and misspellings in English reviews.The filter uses dictionary-based unigram and bigram ratios with preset thresholds.
  • Word stemming and part-of-speech tagging: MARK stems verbs and nouns toward base forms while avoiding the semantic loss caused by aggressive general-purpose stemming.Its rules target selected inflectional suffixes rather than indiscriminately reducing words.
  • Word stemming and part-of-speech tagging: A character-level trigram model detects and repairs over-stemming, changing “analyz” to “analyze” after removing “ing”.

III. KEYWORD RECOMMENDATION

MARK recommends keywords by ranking their association with negative reviews, using contrast score as its main efficient ranking metric. The approach is supported by rating-distribution examples and closely agrees with alternative scores.

  • Keyword ranking: MARK ranks extracted keywords because frequent association with negative reviews may identify app issues or features linked to user dissatisfaction.
  • Keyword ranking: “Drain” appears in 3,060 negative reviews and 726 positive reviews, illustrating the concentration of issue-related keywords in low-rated reviews.
  • Keyword ranking: MARK’s contrast score combines the negative-to-positive review ratio with their absolute count difference.
  • Keyword ranking: Alternative Pearson-correlation and skewness scores capture similar rating-count patterns, such as freeze’s -0.937 correlation and 8.719 skewness.
  • Keyword ranking: The top 1,000 keywords from the three ranking metrics overlap by 96% to 99%, so MARK uses contrast score for efficient incremental computation.

B. Keyword Clustering and Expanding

MARK groups keywords to make analysts’ interest descriptions more cohesive or comprehensive. It uses Word2Vec vectors with k-means clustering or centroid-based expansion to identify related terms.

  • Motivation: Grouping related keywords can improve analysis because users describe the same concern with terms such as “battery”, “power”, “drain”, and “hog”.
  • Grouping functions: MARK clusters a large keyword set into cohesive subsets or expands a small concern-specific set into a broader one.
  • Vector representation: Word2Vec represents review-derived keywords as vectors so syntactically or semantically related terms have similar representations.
  • Keyword clustering: K-means repeatedly assigns each keyword to its nearest centroid and recomputes centroids until cluster assignments stabilize.
  • Keyword expanding: Keyword expansion computes the mean vector of the selected set and adds terms within a predefined distance threshold.

IV. REVIEW SEARCH AND TREND ANALYSIS

MARK searches reviews and analyzes keyword trends using vector-space information retrieval. It ranks reviews by tf.idf cosine similarity and can visualize keyword occurrences over time.

  • Trend analysis: The system also visualizes selected-keyword occurrences over time to help analysts identify unusual patterns or abnormalities.
  • Review search: MARK represents reviews and keyword queries as vectors in a Vector Space Model for relevance-based review search.
  • Review search: The tf.idf weight uses a keyword’s term frequency in a review and its document frequency across the review dataset.
  • Review search: MARK precomputes review vectors, forms a query vector with term frequencies set to 1, and ranks reviews by cosine similarity.Higher cosine scores indicate greater relevance to the keyword query.

B. Trend Analysis

MARK treats keyword occurrences as a time series, smooths short-term noise with a simple moving average, and flags unusually large deviations to detect topic bursts. In the Facebook Messenger energy-consumption example, the method identifies February 12–13, 2015 as sudden changes.

  • Trend Analysis: MARK counts daily occurrences of an analyst-selected keyword set to create a time series for tracking topic activity over time.The example uses energy-consumption keywords for Facebook Messenger from January 1 to April 24, 2015.
  • Trend Analysis: A simple moving average smooths short-term fluctuations so major changes are easier to analyze.The moving average uses the average of the preceding k days and acts like a low-pass filter.
  • Trend Analysis: Comparing actual counts with moving-average values reveals changes that smoothing alone would hide.Sudden changes are identified when actual occurrences substantially exceed the corresponding moving-average value.
  • Trend Analysis: MARK reports a sudden significant change when the relative deviation exceeds 2 standard deviations.The method interprets the moving average as the expected count and compares the difference against the moving-average series’ standard deviation.
  • Trend Analysis: The energy-consumption series exceeded the threshold on February 12 and 13, 2015, matching the burst in keyword occurrences.The trend-analysis figure uses energy-related keywords including battery, drain, overheat, and power, with a ratio threshold of 2.

V. EMPIRICAL EVALUATION

MARK was empirically evaluated on more than two million Google Play reviews from 95 mobile apps collected over five months. The evaluation examined data processing and keyword-analysis components, including English-review classification, with examples drawn from popular apps.

  • Dataset: The evaluation used more than two million reviews collected from Google Play for 95 mobile apps between January 1 and May 1, 2015.Reviews contained titles, text descriptions, creation times, reviewer IDs, and associated ratings.
  • Dataset: The studied apps included popular services and games such as Facebook, Twitter, WhatsApp, Snapchat, Viber, Instagram, and Clash of Clans.Each app averaged about 20,000 reviews, although collection volume varied substantially across apps.
  • Dataset: The collected reviews may not represent all available Google Play reviews because the crawler could not preserve current states or exclude previously crawled results.Google Play returned only 500 reviews per request, requiring continuous crawling over time.
  • English classification: English-review classification achieved a best accuracy of 86.5% on 400 manually labeled reviews.The labeled set contained 245 English and 155 non-English reviews, and the experiment varied unigram and bigram ratio thresholds.
  • English classification: The classifier could recognize English reviews containing words written in another language or unrecognizable words.Examples and corresponding unigram and bigram ratios were presented in Table VIII.

2) Customized Stemming:

MARK evaluates its customized stemmer against manually stemmed review words and the Stanford Lemmatizer. The customized approach achieves higher reported accuracy and is associated with domain-specific spelling and naming resources.

  • Customized Stemming: The stemming evaluation used 1,000 randomly selected verbs and nouns from the review data, manually reduced to their base forms.The manually created forms served as references for comparing MARK’s stemmer with the Stanford Lemmatizer.
  • Customized Stemming: MARK’s stemmer correctly handled 97.9% of words, compared with 90.8% for the Stanford Lemmatizer.The reported improvement is likely attributable to a domain-specific dictionary of misspelled words and special names.

2) Keywords Expanding:

MARK expands keyword sets using vector-based semantic similarity and evaluates the resulting terms through researcher judgments. The expansion technique reaches 89.7% overall accuracy but depends on a distance threshold that can prevent some topics from expanding.

  • Keywords Expanding: The keyword-expansion experiment began with one keyword from each previously reported review topic and had eight researchers assess the expanded results.Researchers identified unrelated keywords using the same evaluation procedure as the clustering experiment.
  • Keywords Expanding: 89.7% overall accuracy was achieved for keyword expansion.Accuracy was computed from researchers’ acceptance judgments for the generated keywords.
  • Keywords Expanding: The results suggest that MARK can capture a keyword set’s semantics and add relevant words to it.The paper notes low accuracy for the “ads” and “call” topics because of misspelled words such as “cal” and “spoor”.
  • Keywords Expanding: Expansion depends on the distance threshold: at 0.2 or lower, some topics could not be expanded because no words met the similarity requirement.The reported expansion experiment used a threshold of 0.25.
  • Review Search: The review-search evaluation returned 90–97% accuracy across the tested top-review cutoffs.The top 10 reviews were mostly relevant, while accuracy decreased as more reviews were included.

E. Threat of Validity

MARK’s validity is constrained by small, potentially unrepresentative datasets and evaluation involving authors and academic participants. Its keyword-based analysis differs from prior review-mining approaches through user-rating-based ranking, semantic keyword grouping, and trend analysis.

  • Threats to Validity: Evaluation used datasets manually labeled by the authors and validated by human subjects, including students and professors rather than intended professional users.The authors identify this evaluation design and participant mismatch as major threats to validity.
  • Threats to Validity: The study’s datasets and experiments were small, covering 95 downloaded apps compared with millions available, and the selected apps might not be representative.These limits constrain how broadly the findings can be generalized.
  • Distinction from Prior Work: MARK ranks keywords using user ratings, providing a domain-specific alternative to general lexical sentiment scoring.Its keywords can also represent sets rather than only two-word collocations, giving users and developers more flexible expression mapping.
  • Distinction from Prior Work: MARK groups keywords with K-means over vector-space representations and detects concern trends regardless of the total number of reviews.This contrasts with prior approaches focused on topic distributions or bursts in positive and negative comments.
  • Distinction from Prior Work: Unlike phrase-level feature extraction, MARK represents app concerns through keywords that can be grouped by semantic similarity and used to retrieve relevant reviews.The framework’s keyword formulation is intended to be more intuitive for developers and users.

VII. CONCLUSION

MARK combines customized review preprocessing with keyword extraction, semantic grouping, relevance search, and temporal anomaly detection. Evaluations and case studies report strong task-specific accuracy and suggest that the framework can help developers find and understand user opinions with less effort.

  • Framework: MARK combines keyword extraction, ranking, semantic grouping, relevance search, temporal visualization, and unusual-pattern reporting in a semi-automated framework.These customized techniques support collecting and mining user opinions from app-market reviews.
  • Preprocessing: 86.5% of reviews were correctly classified as English or non-English in the test set, while the customized stemmer outperformed Stanford’s general-purpose lemmatization tool on app-review data.The conclusion attributes these preprocessing components to the difficulties of handling raw reviews.
  • Semantic Keyword Analysis: 83% grouping accuracy and 89.7% expanding accuracy were reported for the semantic keyword techniques in case studies.The techniques use distributed vector-space representations to identify related keywords expressing common concerns.
  • Review Search: 90% of the top 50 reviews returned for a Facebook Messenger battery-consumption query satisfied the query in the case study.The search uses keywords discovered through the framework’s preceding analysis steps.
  • Trend Analysis: The moving-average analysis correctly detected a real Facebook Messenger problem in a case study and was judged to have potential to reduce developers’ effort.The reported abnormal-pattern detection concerns keyword occurrences over time.
  • Practical Implication: The authors suggest MARK can help developers map their concerns to users’ keyword expressions, saving time and effort when discovering and understanding opinions.This practical implication is based on the reported evaluations and case studies.
Loading 1505.04657v2…