Source-linked AI summary
Fighting Authorship Linkability with Crowdsourcing
Mishari Almishari, Ekin Oguz, Gene Tsudik
TL;DR
Simple stylometric features can link reviews across accounts, raising privacy concerns. The paper improves review linkage and tests crowdsourced rewriting and translation as mitigation approaches. Crowdsourced rewriting reduces linkability, and crowdsourcing can improve translated reviews while retaining low linkability.
Problem
Simple stylometric features can link reviews by the same person across accounts, posing a privacy concern.
Source: §1 · pp. 1–2Method
The paper improves review-linkage methods, then studies crowdsourced rewriting, translation, and crowdsourcing to fix poorly readable translations.
Source: §1 · p. 2, §8 · pp. 12–13Results
Crowd rewrites reduced Top-1 LR from 95% to 55% among 40 authors and from 77.5% to 10% among 1000 authors; crowdsourcing also fixed poorly readable translations while retaining low linkability.
Source: §6.1.6 · p. 8, §8 · p. 13Takeaways & Limitations
Crowdsourced rewrites remained meaningful and similar to originals while decreasing linkability.
Source: §8 · p. 12Further work is needed on other review sites, cross-site studies, and detailed stylometric evaluation.
Source: §8 · p. 13
Abstract
from arXiv · showhide
Massive amounts of contributed content -- including traditional literature, blogs, music, videos, reviews and tweets -- are available on the Internet today, with authors numbering in many millions. Textual information, such as product or service reviews, is an important and increasingly popular type of content that is being used as a foundation of many trendy community-based reviewing sites, such as TripAdvisor and Yelp. Some recent results have shown that, due partly to their specialized/topical nature, sets of reviews authored by the same person are readily linkable based on simple stylometric features. In practice, this means that individuals who author more than a few reviews under different accounts (whether within one site or across multiple sites) can be linked, which represents a significant loss of privacy. In this paper, we start by showing that the problem is actually worse than previously believed. We then explore ways to mitigate authorship linkability in community-based reviewing. We first attempt to harness the global power of crowdsourcing by engaging random strangers into the process of re-writing reviews. As our empirical results (obtained from Amazon Mechanical Turk) clearly demonstrate, crowdsourcing yields impressively sensible reviews that reflect sufficiently different stylometric characteristics such that prior stylometric linkability techniques become largely ineffective. We also consider using machine translation to automatically re-write reviews. Contrary to what was previously believed, our results show that translation decreases authorship linkability as the number of intermediate languages grows. Finally, we explore the combination of crowdsourcing and machine translation and report on the results.
1. INTRODUCTION
Community reviews are widely shared, and simple stylometric features can link reviews across accounts, raising privacy concerns. The paper develops a stronger linkage technique and explores crowdsourcing and translation to mitigate linkability.
1. INTRODUCTION
- Simple stylometric profiles can link reviews from different accounts to the same contributor, creating a privacy concern even though tracking spam reviews may be useful.The paper presents harassment by a vindictive merchant as one example of the privacy risk.Source: §1 · pp. 1–2
- The authors improve review-linkage methods, reporting 90% accuracy with 95 identified reviews and only 5 anonymous reviews.The technique is intended to assess the efficacy of proposed mitigation methods.Source: §1 · p. 2
- The paper tests crowdsourced rewriting and machine translation as ways to decrease authorship linkability.Crowdsourced rewrites are reported as meaningful and generally faithful; randomly selected translation languages also decrease linkability.Source: §1 · p. 2
2. RELATED WORK
Related work covers authorship attribution and anonymization. The paper distinguishes its review-linkage setting and explores translation variations beyond prior round-trip approaches.
2. RELATED WORK
- Prior work links Yelp reviews using simple features, while studies report 80% linkability for blogs and 90% accuracy for academic reviews.Reviews are shorter, less formal, and less restrictive in word choice than blogs and academic papers.Source: §2 · p. 2
- Writeprint features have been used for author attribution, while this study focuses on short reviews and linkage with few reviews per author.The section also cites a comprehensive overview of authorship attribution studies.Source: §2 · p. 2
- Earlier round-trip translation studies used at most two intermediate languages and found no noticeable anonymizing effect; this work varies their number and selection.Rao and Rohatgi were among the first to propose round-trip translation for obfuscating authors.Source: §2 · p. 2
- Feature-adjustment anonymization may require a large text corpus, and Anonymouth requires additional same-author and other-author document sets [19], unlike this approach.Stylistic deception methods have also been detected with 96.6% accuracy.Source: §2 · pp. 2–3
3. BACKGROUND
This section introduces the stylometric features and statistical distance used to analyze review authorship. The study uses Writeprints features and Chi-Squared distance to compare review distributions.
3. BACKGROUND
- Stylometry concerns recurring expression patterns, and this study uses Writeprints features and a Chi-Squared test to assess review linkability.The test computes distances between anonymous and identified reviews.Source: §3 · p. 3
3.1 Writeprints
- Writeprints combines static and dynamic features capturing lexical, syntactic, structural, content, and idiosyncratic properties of text.Examples include character-per-word counts, letter trigrams, POS tag bigrams, and function words.Source: §3.1 · p. 3
3.2 Chi-Squared Test
- The Chi-Squared test measures distance between distributions; its distance is symmetric, non-negative, and zero when the distributions are identical.The study applies it to distributions from contributors’ anonymous and identified reviews.Source: §3.2 · p. 3
4. LINKABILITY STUDY PARAMETERS
The linkability study uses Yelp reviews and defines a task that matches anonymous review sets to identified records. It varies the anonymous-set size while keeping the records small relative to the dataset’s average author history.
4. LINKABILITY STUDY PARAMETERS
- The study parameters define the dataset and problem setting for review-linkability analysis.The setup treats each anonymous review set as a candidate match to identified reviews with a known author.Source: §4 · p. 3, §4.2 · p. 3
4.1 Dataset
- The Yelp dataset contains 1,076,850 reviews from 1,997 contributors, averaging 539 reviews per author and 149 words per review.The authors selected it for its large, varied author histories and relatively short reviews.Source: §4.1 · p. 3
4.2 Problem Setting
- For each of 40 randomly selected authors, the study sets N=100 reviews and varies the anonymous record size X from 1 to 5, matching it to the remaining identified reviews.A correct author within the top T ranked candidates counts as a hit; the study considers T=1 and T=4.Source: §4.2 · pp. 3–4
5. LINKABILITY ANALYSIS
The linkability analysis develops a review-linkage model using stylometric features and Chi-Squared distances, then selects features that sustain high linkage rates as the candidate-author set grows.
5.1 Methodology
- Review sets are tokenized by selected features, compared using Chi-Squared distances, and ranked by ascending distance to identify the most likely match.The analysis repeats the experiment three times with randomly selected sets and averages the results.Source: §5.1 · p. 4, §5 · p. 4
5.2 Feature Selection
- A greedy hill-climbing heuristic adds influential features to the selected set until it encounters a high linkability rate.Source: §5.2 · p. 4
5.2.1 WPall
- With |AR|=5, WPall reaches 52.5% Top-1 and 82.5% Top-4; the authors attribute its lower rates to noise from combining many features.Source: §5.2.1 · p. 4
5.2.2 Improving WPall
- Top Letter Trigrams raises Top-1 linkability from WPall’s 52.5% to 91%, and ranks first among the individual features.The Table 2 caption identifies the comparison as the five best individual Writeprint features against WPall at |AR|=5.Source: §5.2.2 · p. 4, §5.2.1 · p. 4, Table 2 · p. 4
5.2.3 Improving Top Letter Trigrams
- Adding POS Bigrams to Top Letter Trigrams raises linkability from 91% to 96% Top-1 and from 96% to 100% Top-4.Source: §5.2.3 · p. 4
- Across AR sizes, Top Letter Trigrams outperforms Writeprints, and adding POS Bigrams further improves both Top-1 and Top-4 linkability.Source: §5.2.3 · p. 4
5.3 Scalability of the Linkability Technique
- With |AR|=5, the selected pair achieves 77.5% Top-1 and 90% Top-4 among 1,000 possible authors.The figure compares linkability rates for Writeprints, Letter3, and Letter3+POS2.Source: Figure 1 · p. 5, §5.3 · p. 5
5.4 Summary
- Top Letter Trigrams plus POS Bigrams reaches 96% Top-1 and 100% Top-4; among 1,000 possible authors, it achieves 77.5% Top-1 and 90% Top-4.The selected pair is used to evaluate anonymization techniques.Source: §5.4 · pp. 5–6
6. FIGHTING AUTHORSHIP LINKABILITY
This section examines crowdsourced rewriting, machine translation, and crowdsourcing-based fixing of translated reviews as approaches to reduce authorship linkability, including their effects on review meaning and readability.
6.1 Crowdsourcing to the Rescue
- Because topic and temporal clues can still link reviews, the authors target stylometric linkability; observed crowd rewrites changed sentence structure and substituted synonyms, while an automated rewriting tool was left for future work.They describe perfect anonymization as probably impossible and did not pursue a software tool that could flag or suggest changes, citing complexity and evaluation difficulties.Source: §6.1 · p. 6, §6.1.1 · p. 6, §6.1.5 · p. 8
6.1.3 Rewriting Phase
- Five MTurk workers rewrote each of 200 original reviews; 882 of 1,000 submissions were accepted, and each 250-word review cost $0.12 on average.Workers were instructed to preserve meaning and proper names, and submissions far from the original’s word count were rejected; most rewrites arrived within 3–4 days.Source: §6.1.3 · pp. 6–7, §6.1.6 · p. 8
6.1.4 Readability Study
- Crowdsourced rewrites averaged 4.29/5 for readability, and 87% received Good or Excellent scores, indicating that they generally retained the originals’ meaning.The assessment used two MTurk raters per original–rewrite pair.Source: Figure 3 · p. 7, §6.1.4 · p. 7
6.1.5 Linkability of Rewritten Reviews
- For |AR| = 5, rewrites reduced Top-1 LR from 95% to 55% in a 40-author set and from 77.5% to 10% in a 1,000-author set; in 1,000 authors, Top-4 fell from 90% to 32.5%.Source: §6.1.5 · p. 7, §6.1.6 · p. 8
6.2.1 Translation Framework
- The translation framework randomly selected intermediate languages, translated reviews through them in sequence, and then translated back into English; the study varied the count from 1 to 9.Randomizing language sequences was intended to make it less likely that reviews would retain stylometric patterns.Source: §6.2.1 · p. 9, §6.2.2 · p. 9
6.2.2 Linkability of Translated Reviews
- For |AR| = 5, translation with 9 intermediate languages yielded Top-1 and Top-4 LRs of 25% and 40%, versus 77.5% and 90% for original reviews.With 4 and 7 languages, the respective Top-1/Top-4 LRs were 42.5%/59% and 31%/47%; LRs declined as the number of intermediate languages increased.Source: §6.2.2 · pp. 9–10, Figure 4 · p. 10
6.2.3 Readability of Translated Reviews
- Translated reviews averaged 2.85/5 for readability, with most scores at least Fair; the authors then asked MTurk workers to fix unreadable parts.They identified awkward proper nouns, sentence organization, and untranslated terms as common problems; fixing was presented as easier than rewriting the whole review.Source: §6.2.3 · p. 10, Figure 5 · p. 11, §6.2.4 · p. 11
6.2.4 Fixing the Translated Reviews
- For AR-5, fixing translated reviews left Top-1 LR at 19%, versus 25% for unfixed translations and 74% for originals.The authors report that fixing translations did not significantly influence linkability; only 31 authors had all their translated reviews fixed.Source: §6.2.4 · p. 11
- Average readability score increased from 2.85 to 4.12 after crowdsourcing fixed machine translations, whose counterparts mostly scored Good or Excellent.The authors say the fixes preserve unlinkability while correcting the meaning of machine-translated reviews.Source: §6.2.4 · p. 11, Figure 7 · p. 11
6.2.5 Comparison of Anonymization Techniques
- Linkability fell substantially across author-set sizes; rewritten reviews had the lowest linkability, while translated and translated-fixed reviews performed comparably.Crowdsourcing had a mean readability score of 4.29/5, compared with 2.85/5 for translation and 4.12/5 for fixed translations.Source: §6.2.5 · p. 11
- Crowdsourcing offers better anonymity and readability but takes longer than automated, cost-free machine translation, which has poor readability.Crowdsourcing is not free, though the paper describes it as not expensive.Source: §6.2.5 · p. 11
7. DISCUSSION
The discussion weighs crowdsourcing’s usefulness against its practical constraints and open questions. The authors consider its applicability, latency, cost, and privacy risks.
7. DISCUSSION
- The study uses Yelp, while applying crowdsourced rewriting to other OSNs is presented as a possibility rather than an evaluated result.The authors cite activist blogs, Twitter, and TripAdvisor as potential settings.Source: §7 · pp. 11–12
- A delay of a couple of days is considered tolerable because review posting need not be real-time and site moderation can take weeks.TripAdvisor is given as an example of moderation taking several weeks.Source: §7 · p. 12
- US$0.12 was the average cost per rewriting task, which the authors consider potentially subsidizable by plug-in advertising.Source: §7 · p. 12
- The authors say privacy risk is difficult to assess unless an adversary both works on a task and gets the rewritten submission published.They add that tracking many posted reviews would not scale for an adversary and would require manual follow-up.Source: §7 · p. 12
- The authors expect worker-style recognition to be mitigated by selecting among many workers and limiting repeated assignments.They also expect workers to rewrite reviews from different sources, widening topic coverage; worker identities are expected to remain private to the requester.Source: §7 · p. 12
8. CONCLUSIONS AND FUTURE WORK
The paper reports that crowdsourced rewriting and machine translation reduce review linkability, while rewriting produces more readable results than translation. It also identifies further evaluation and broader review-site studies as future work.
8. CONCLUSIONS AND FUTURE WORK
- Crowdsourced rewriting substantially decreases linkability while producing reviews that remain meaningful and similar to the originals.Source: §8 · p. 12
- Linkability decreases as the number of intermediate translation languages increases.Source: §8 · p. 12
- Online-translated reviews are less readable than rewritings, while crowdsourcing can improve translation readability and retain low linkability.Source: §8 · p. 13
- Future work calls for more sophisticated evaluation of stylometric differences and studies on other review sites and across sites.Examples include Amazon, TripAdvisor, and Ebay, as well as Amazon-Yelp combinations.Source: §8 · p. 13
APPENDIX Appendix A: Crowdsourcing Examples
The appendix presents example submissions from the MTurk rewriting and readability tasks and points readers to the full collection.
APPENDIX Appendix A: Crowdsourcing Examples
- The appendix presents example submissions from both the rewriting and readability tasks.Source: APPENDIX Appendix A: Crowdsourcing… · p. 14
- The full collection of original and rewritten reviews is available in.Source: APPENDIX Appendix A: Crowdsourcing… · p. 14
A.1: Rewriting Example
The rewriting example recounts a long wait, poor seating and service, and a meal the reviewer ultimately describes as delicious.
A.1: Rewriting Example
- The reviewer describes arriving to find a line around the block and being seated at the back of the establishment.Source: §A.1 · p. 14
- The review describes delayed assistance and food arriving after the reviewer had waited half an hour for staff.Source: §A.1 · p. 14
- The reviewer calls the food delicious, compares the eggplant unfavorably with China Garden, and closes with “Authentic and super.”Source: §A.1 · p. 14
A.2 Readability Example
A sample readability-study submission notes that the alternative review preserves the main points but changes some details and tone.
A.2 Readability Example
- The reader found the reviews similar on main points, but noted an omitted “pretty” and a more negative description using “smug.”Source: §A.2 · p. 14
Appendix B: Translation Examples
The appendix gives examples of round-trip machine translations through different numbers of intermediate languages, a translation-fixed version, and MTurk tasks.
Appendix B: Translation Examples
- The examples show round-trip translations using three, six, and nine intermediate languages, followed by an MTurk-fixed version of the nine-language translation.Source: Appendix B · pp. 14–15
- The full collection of translated reviews is available in.Source: Appendix B · p. 15
- Figures 8 and 9 illustrate MTurk tasks, including a sample translation-fix task.Source: Figure 8 · p. 16, Figure 9 · p. 17