PaperlayerView exact revision on arXiv

Source-linked AI summary

Battling the Internet Water Army: Detection of Hidden Paid Posters

Cheng Chen, Kui Wu, Venkatesh Srinivasan, Xudong Zhang

arXiv:1111.4297v1cs.SI

TL;DR

The paper asks how hidden paid posters can be distinguished from legitimate online users despite limited public behavioral data. It analyzes real-world traces using non-semantic and semantic features, and reports promising detection performance, including improved precision, recall, F-measure, and accuracy.

  • Problem

    Paid posters can spread untrustworthy or misleading information, but systematic study is hindered because they work underground and public behavioral data are unavailable.

  • Method

    The authors collect real-world datasets, analyze four non-semantic behavioral features, and integrate semantic analysis into a detection model.

  • Results

    Precision, recall, F-measure and accuracy improved to 95.24%, 73.17%, 82.76% and 88.79%, respectively, with semantic analysis.

  • Takeaways & Limitations

    Semantic features supplement behavioral features because repeated comments with minor edits help the classifier detect potential paid posters.

Abstract

from arXiv · show

We initiate a systematic study to help distinguish a special group of online users, called hidden paid posters, or termed "Internet water army" in China, from the legitimate ones. On the Internet, the paid posters represent a new type of online job opportunity. They get paid for posting comments and new threads or articles on different online communities and websites for some hidden purposes, e.g., to influence the opinion of other people towards certain social events or business markets. Though an interesting strategy in business marketing, paid posters may create a significant negative effect on the online communities, since the information from paid posters is usually not trustworthy. When two competitive companies hire paid posters to post fake news or negative comments about each other, normal online users may feel overwhelmed and find it difficult to put any trust in the information they acquire from the Internet. In this paper, we thoroughly investigate the behavioral pattern of online paid posters based on real-world trace data. We design and validate a new detection mechanism, using both non-semantic analysis and semantic analysis, to identify potential online paid posters. Our test results with real-world datasets show a very promising performance.

I. INTRODUCTION

The paper studies hidden online paid posters, whose coordinated activity can influence online opinion and undermine trust in Internet information. It proposes combining behavioral and semantic analysis to detect potential paid posters using real-world data.

  • Motivation: Online paid posters are a growing occupation in which users follow company instructions to post comments and articles for hidden purposes.They are also called the “Internet water army” in China and are often college students or unemployed people.
  • Motivation: Paid posters can support marketing but may also spread fake or negative information about competitors, creating difficulties for ordinary users seeking trustworthy information.The paper notes that immature laws and supervision mechanisms may allow such activity without penalties.
  • Research gap: The paper addresses a lack of systematic research caused by paid posters working underground and the absence of public behavioral data.It presents the work as the first study tackling detection of potential paid posters.
  • Contributions: The authors collect real-world event data, analyze behavioral patterns, identify useful features, and integrate semantic analysis to improve detection accuracy.The contributions include confirming the organizational structure of paid posters through first-hand participation.

II. HOW DO ONLINE PAID POSTERS WORK?

The paper illustrates how paid posters can be organized to amplify positive marketing or negative claims through coordinated activity across online communities. It uses documented social events and direct participation in a paid-posting website to characterize this work.

  • Typical Cases: Paid-posting campaigns may promote a company or disseminate negative claims, as illustrated by the conflict involving 360 and Tencent.Articles alleging privacy violations spread rapidly across websites and attracted large numbers of views and replies.
  • Case selection: The case study focuses on the 360–Tencent conflict because large amounts of related comments and articles remained available on popular websites.The authors use this event as the basis for subsequent data collection.
  • Work process: Paid posters register with banking accounts, receive missions and posting instructions, and are rewarded after assignments pass validation.Missions can include posting advertisements, comments, videos, or carrying out question-and-answer sessions.
  • Work process: Because rewards are usually based on the number of posts, posters may not strictly follow mission rules, producing behavioral patterns that can be analyzed statistically.Rules include posting across forums, avoiding copied replies, and completing missions on time.

2) Management of Paid Posters:

Paid-posting missions are coordinated through a PR-company structure that divides planning, posting, relationship management, and resource provision among specialized teams.

  • Management structure: Each mission typically assigns one project manager and four teams: trainer, poster, public relationship, and resource teams.The authors’ first-hand experience confirms an organizational structure similar to one previously disclosed.
  • Management structure: The project manager coordinates the activities of the four teams throughout the mission.
  • Team roles: The trainer team schedules posting activities and distributes shared user IDs, while the posters team publishes information for payment.Posters are often college students or unemployed people and may be grouped by target website or community.
  • Team roles: The public relationship team maintains relationships with webmasters to prevent deletion and may encourage them to highlight posts.
  • Team roles: The resources team supplies user IDs, registration information, and prepared post templates, including templates written by hired writers.

III. DATA COLLECTION

The authors build two real-world datasets from Sina.com and Sohu.com about the 360–Tencent conflict, using Sina for training and Sohu for testing. They address dynamic pages, duplicate records, and substantial differences between the datasets.

  • Data sources: The study collects news reports and comments about the 360–Tencent conflict from Sina.com and Sohu.com.Sina is used as training data, while Sohu is used as test data.
  • Data collection: The search covers September 10 to November 21, 2010, yielding 22 Sina news reports and 24 Sohu news reports.Recorded comment fields include report ID, sequence number, post time, location, user ID, content, and response indicator.
  • Data cleaning: Dynamic web pages required a crawler capable of obtaining comments through additional server requests generated by AJAX.The authors adopted Gooseeker to retrieve the dynamically generated data.
  • Data cleaning: After duplicate removal, the datasets contained 53,723 Sina records and 115,491 Sohu records.
  • Dataset characteristics: The final Sina dataset contained 552 users and 20,738 comments, whereas Sohu contained 223 users and 1,220 comments.Average comments per user were about 37.6 for Sina and 5.5 for Sohu, partly because Sohu allows anonymous posts.
  • Evaluation question: A central evaluation question is whether a detector trained on one dataset can work effectively on another dataset.

IV. NON-SEMANTIC ANALYSIS

The paper uses four behavioral features to distinguish potential paid posters from normal users, beginning with manually selected candidates and statistical analysis of their posting behavior.

  • Ground truth and features: 70 potential paid posters were manually selected from 552 users to establish a cautious analysis set.The paper uses “potential” because manual selection cannot establish with certainty that users are paid posters.
  • Ground truth and features: The analysis tests reply percentage, average posting interval, active days, and the number of news reports commented on.The first two features are later identified as showing strong differences between the groups.
  • A. Percentage of Replies: 84.3% of potential paid posters had fewer than 50% of posts as replies, whereas 73.2% of normal users posted more replies than new comments.The reply ratio p is the number of replies divided by a user’s total comments.
  • A. Percentage of Replies: Potential paid posters were more likely to create new comments rather than read and reply to others’ comments.

B. Average Interval Time of Posts

Average posting intervals and active-day duration reveal behavioral differences between potential paid posters and normal users, although some potential paid posters exhibit slower posting patterns.

  • B. Average Interval Time of Posts: The average interval is computed within active epochs separated by breaks exceeding 24 hours, then averaged across epochs.This reduces the effect of long breaks before a user resumes posting.
  • B. Average Interval Time of Posts: 50% of potential paid posters posted with intervals below 2.5 minutes, compared with 23% of normal users.Nearly 80% of potential paid posters had intervals below 5 minutes, compared with 57% of normal users.
  • B. Average Interval Time of Posts: Potential paid posters were more likely to post within very short time periods, consistent with completing assignments quickly rather than joining discussion.
  • B. Average Interval Time of Posts: Some potential paid posters posted slowly, with intervals longer than 750 seconds.The paper attributes these outliers partly to trainer-enforced rules intended to reduce detection.
  • C. Active Days: No potential paid posters remained active for more than four days, while about 4% of normal users participated for five or more days.The groups were otherwise nearly identical for one through four active days.

D. The Number of News Reports

The number of news reports commented on does not clearly distinguish potential paid posters from normal users in the Sina dataset.

  • D. The Number of News Reports: Potential paid posters and normal users had similar distributions for the number of news reports they commented on.
  • D. The Number of News Reports: The Sina results did not provide strong evidence that paid posters comment on more news reports than normal users.This contradicted the paper’s initial conjecture about broader commenting across reports.
  • D. The Number of News Reports: The number of commented news reports alone may not be a good feature for detecting potential paid posters.

E. Other Observations

The paper examines additional behavioral signals for identifying potential paid posters, including geographic anomalies, contradictory comments, and irrelevant replies. It also introduces semantic similarity analysis, which distinguishes potential paid posters from normal users in the Sina dataset.

  • Geographic patterns: Geographic distribution does not clearly distinguish potential paid posters from normal users, who are both concentrated in central and southern China.The locations of potential paid posters were also unrelated to the locations of Tencent and 360.
  • Geographic patterns: A user ID appearing in different geographic locations within minutes or hours is proposed as a clear paid-poster indicator, but the Sina dataset lacked sufficient evidence.The paper attributes this possibility to dynamically assigned user IDs.
  • Comment behavior: Contradictory comments may indicate paid posters, but detecting this feature requires understanding comment meaning and remains challenging to implement.The proposed explanation is that posters publish content without personal emotion or careful checking.
  • Comment behavior: Irrelevant replies may result from paid posters copying existing posts and clicking reply without reading the original content, but this feature is hard to implement.The paper links this behavior to increasing the total number of posts for additional payment.
  • Semantic analysis: The semantic method compares content-word lists after Chinese word segmentation and flags comment pairs whose shared-word ratio exceeds a threshold.Stop words are filtered, and a threshold of three similar comment pairs reduces false signals from normal users who accidentally duplicate comments.
  • Semantic analysis: 79.6% of normal users had no similar comment pairs, whereas 78.6% of potential paid posters had more than 5 similar pairs in the Sina dataset.These statistics support similarity between comments as a useful detection feature for potential paid posters.

VI. CLASSIFICATION

The classification system models detection as a binary classification problem, combining investigated behavioral features and training an SVM on the Sina dataset. It evaluates the resulting classifier on Sohu using manually identified labels and standard classification metrics.

  • Classification system: The system classifies each user as a potential paid poster or normal user using combinations of five statistical and semantic features.The paper reports that no single feature is sufficient, motivating comparison of feature combinations.
  • Classification system: An SVM performs the binary classification, with LIBSVM, a radial basis function, and 10-fold cross-validation used for training and testing.The classifier trained on Sina data is tested on the Sohu dataset.
  • Evaluation: The Sohu test set is manually labeled by reading users’ posts before classifier evaluation.The manually selected paid posters and normal users are listed in Table VII.
  • Evaluation: Performance is evaluated with precision, recall, F-measure, and accuracy, using manual identification as the benchmark result.These are presented as standard classification-system metrics.
  • Evaluation: The paper also reports the number of similar comment pairs posted by each user as part of the semantic-analysis evaluation.Figure 7 visualizes the corresponding statistic.

A. Classification without Semantic Analysis

Classification using only statistical features performs poorly because the first two features introduce substantial bias. Adding four non-semantic features improves results, but false positives, false negatives, precision, and accuracy remain problematic, motivating semantic analysis.

  • Two-feature classification: For the 2-feature test, precision was 100%, but the classifier correctly identified only 2 of 82 potential paid posters.The paper considers this performance unacceptable for finding paid posters.
  • Two-feature classification: The first two features produced significant classification bias, indicating that additional features were necessary.The two features were ratio of replies and average interval time of posts.
  • Four-feature classification: Using four non-semantic features improved most results except precision and accuracy, but false positives and false negatives remained too high for acceptable performance.The resulting low precision made the classifier unreliable.
  • Four-feature classification: The paper improves the classifier further by adding semantic analysis to the non-semantic feature set.This follows the reported unreliability of classification based on four non-semantic features alone.

B. Classification with Semantic Analysis

The paper augments behavioral detection of online paid posters with semantic analysis, exploiting similarities among their comments. Integrated into an SVM classifier, this approach substantially improves detection performance on the Sohu dataset.

  • Classification with Semantic Analysis: Semantic analysis was integrated into the SVM model to supplement behavioral features for detecting online paid posters.The method was tested again on the Sohu dataset after integration.
  • Classification with Semantic Analysis: 95.24% precision, 73.17% recall, 82.76% F-measure, and 88.79% accuracy were achieved after adding semantic analysis.These results are reported for the Sohu dataset.
  • Classification with Semantic Analysis: Online paid posters often make minor edits to repeated comments, producing similar sentences that help classifiers identify them.The repetition also helps posters complete assignments quickly.
  • Study Scope: The study characterizes paid posters’ organizational structure and posting patterns in the context of influencing opinions about popular social events.The authors describe the work as the first study of the social phenomenon of paid posters, to their knowledge.
  • Relation to Prior Detection Work: The work positions paid-poster detection alongside prior research on forum, blog, video-response, and social-network spam detection.Prior studies examined behavioral attributes, content, comment similarity, and spam campaigns.
Loading 1111.4297v1…