Evaluating user reputation in online rating systems via an iterative group-based ranking method

1. Introduction

• Noisy and malicious ratings can distort object reputation and make it urgent to measure user credibility and filter untrusted users.The paper cites poor judgment and deliberate maximal or minimal ratings as sources of harmful noise.Source: §1 · p. 1
• Network-based methods efficiently allocate resources in bipartite networks but suffer from rating noise and have limited performance for user reputation evaluation.Source: §1 · p. 1
• Quality-based methods estimate object quality and assess users by rating deviations, but some may not converge or resist spamming attacks robustly.Source: §1 · p. 1
• Quality-based methods also assume each object has one most objective rating, despite subjective ratings allowing multiple reasonable evaluations.Source: §1 · p. 2
• GR groups users by rating similarity and uses group sizes for reputation, but its one-step, equal-contribution process is vulnerable to many large-degree spammers.Source: §1 · p. 2
• IGR introduces iterative reputation allocation: high-reputation users receive greater weights in group sizes, improving performance and robustness over GR on MovieLens and Netflix with artificial spammers.Source: §1 · p. 2

2. Methods

• The rating system is modeled as a weighted bipartite network whose users, objects, and ratings form a rating matrix with discrete values.Source: §2 · p. 2

2.1. Group-based ranking methods

• IGR starts users with equal reputation, groups users by shared ratings, weights group sizes by reputation, normalizes those sizes, and maps ratings into rewards.Source: §2.1 · p. 2
• IGR reallocates reputation from rewarding vectors, penalizing ratings that deviate from group consensus or vary substantially.Source: §2.1 · p. 2
• The reputation and group-size updates continue until the reputation change is below ∆ = 10^-4; without iteration, IGR degenerates to GR.Source: §2.1 · p. 2

2.2. Quality-based ranking methods

• Quality-based methods estimate each object's weighted-average quality and evaluate users against those estimates, including IR, CR, and RR.Source: §2.2 · p. 3
• IR iteratively updates reputation from rating-quality differences, whereas RR starts from activity-based reputation and redistributes nonnegative rating-quality correlations.Source: §2.2 · p. 3

3.1. Real rating data

• MovieLens contains 100000 ratings from 943 users on 1682 movies, while Netflix contains 197248 ratings from 3000 users across 2779 movies.Both use five-point movie-rating scales; MovieLens has larger average user degree, smaller average object degree, and higher sparsity than Netflix.Source: §3.1 · p. 4

3.2. Artificial rating data

• Artificial data replace ratings from selected users with malicious extreme ratings or random ratings to create controlled spammer populations.Malicious spammers receive ratings of 1 or 5 with equal probability, while random spammers receive uniformly selected integers from 1 to 5; the spammer ratio is p = d/m.Source: §3.2 · p. 4

3.3. Evaluation metric

• Recall measures the fraction of artificial spammers detected among the top-L users, with L = d making recall equivalent to precision.Higher recall indicates more accurate ranking, and d′(L) is the number of detected spammers in the top-L list.Source: §3.3 · p. 4
• AUC measures the probability that a randomly chosen spammer receives lower reputation than a randomly chosen normal user, with values above 0.5 indicating better-than-random ranking.AUC is independent of the selected list length and approaches 0.5 for random ranking.Source: §3.3 · p. 4

3.4. Self-consistency metric

• Self-consistency is assessed through the correlation between user reputation and rating error relative to estimated object quality.Rating error measures deviation between a user’s ratings and estimated object qualities; higher correlation indicates greater self-consistency.Source: §3.4 · p. 4

4.1. Reputation evaluation

• Across reputation distributions, CR has the highest diversity, while GR and IGR have nearly equal diversity near 0.90 for MovieLens and 0.88 for Netflix.IR is Poisson-like, CR, GR, and IGR are normal-like, and RR is exponential-like with many users receiving zero reputation.Source: §4.1 · p. 5, Figure 2 · p. 5
• GR and IGR show the strongest self-consistency because reputation is high for low rating error and stably low for high rating error.Their Pearson correlations between reputation and rating error are −0.8166 and −0.8201 for MovieLens, and −0.7353 and −0.7629 for Netflix.Source: §4.1 · p. 5, Figure 3 · p. 6, Table 2 · p. 6
• IR favors high-activity users, whereas CR, GR, and IGR show no obvious degree preference and GR and IGR have weak positive relations with trend following.IR correlates positively with user degree and negatively with trend following; GR and IGR correlate weakly positively with trend following.Source: §4.1 · pp. 5–6, Table 2 · p. 6

4.2. Random spamming analysis

• Against random spamming, GR and IGR perform best in top-rank recovery, while their AUC remains around 0.96 and exceeds the other methods.IGR is reported as more robust than GR, and group-based methods outperform quality-based methods overall.Source: §4.2 · pp. 6–7
• For random spammers grouped by degree, IGR and GR outperform other methods for low-degree spammers, while the overall ranking is IR, RR, CR, GR, then IGR from worst to best.The comparison divides users into Low, Mid, and High degree groups and averages AUC over 100 independent realizations.Source: §4.2 · p. 7, Figure 6 · p. 8

4.3. Malicious spamming analysis

• For malicious spammers, GR and IGR perform best when the spammer ratio is small, with GR degrading faster than IGR as that ratio increases.At small ratios, Rc for GR and IGR is around 0.8, whereas CR and RR are almost zero.Source: §4.3 · p. 8
• IGR achieves the best overall performance against malicious spamming, with AUC values over 0.95, while remaining more robust than GR as the spammer ratio increases.For all malicious spammers, IGR again ranks best; group-based methods outperform quality-based methods overall.Source: §4.3 · pp. 8–9
• In malicious-spamming degree groups, GR and IGR perform best for low-degree spammers, although IGR is best when all spammers are ranked.IR performs better for mid- and high-degree spammers, while CR and GR perform better for high-degree spammers.Source: §4.3 · p. 9, Figure 7 · p. 9

• Ratings from high-reputation users receive higher weights in group-size calculations, while reputation and group sizes are iteratively updated until stable.Source: §5 · p. 9
• Experiments on two real data sets suggest that IGR remarkably outperforms previous quality-based methods.Source: §5 · p. 9
• IGR is more robust than the original GR method because poorly reputed users have less influence on large groups and spammer reputation decays across iterations.Source: §5 · pp. 9–10
• Group-based methods consider users’ grouping behaviors, whereas quality-based methods estimate objects’ true qualities.Source: §5 · p. 9
• The proposed method is reported to improve accuracy and robustness while being easier to implement.Source: §5 · p. 10
• Theoretical convergence remains insufficiently justified, and the effects of continuous versus discrete rating scales remain open questions.Source: §5 · p. 10