Source-linked AI summary
How Community Feedback Shapes User Behavior
Justin Cheng, Cristian Danescu-Niculescu-Mizil, Jure Leskovec
TL;DR
Although ratings are widely studied at the community level, less is known about how they affect the authors whose content is rated. Comparing rewards and punishments across future behavior, the paper finds that negatively evaluated authors’ later posts decline in quality, while rewarded authors’ posts do not improve.
Problem
Although prior work studied ratings at the community level, little attention has been devoted to their effects on evaluated authors, and empirical evidence about operant conditioning in complex online communities is limited.
Source: Introduction · p. 1Method
The paper develops a methodology to compare how rewards and punishments affect multiple facets of authors’ future behavior in communities.
Source: Introduction · p. 1Results
After negative evaluation, authors’ subsequent posts decline significantly in textual quality and are perceived as lower quality, while positively evaluated authors’ posts do not improve significantly.
Source: Summary · p. 6Takeaways & Limitations
In these communities, ratings do not fulfill the predicted role of guiding authors toward better future content.
Source: Summary · p. 6Textual quality is difficult to estimate, and the validated regression model’s goodness of fit could be improved.
Source: Discussion · p. 9
Abstract
from arXiv · showhide
Social media systems rely on user feedback and rating mechanisms for personalization, ranking, and content filtering. However, when users evaluate content contributed by fellow users (e.g., by liking a post or voting on a comment), these evaluations create complex social feedback effects. This paper investigates how ratings on a piece of content affect its author's future behavior. By studying four large comment-based news communities, we find that negative feedback leads to significant behavioral changes that are detrimental to the community. Not only do authors of negatively-evaluated content contribute more, but also their future posts are of lower quality, and are perceived by the community as such. Moreover, these authors are more likely to subsequently evaluate their fellow users negatively, percolating these effects through the community. In contrast, positive feedback does not carry similar effects, and neither encourages rewarded authors to write more, nor improves the quality of their posts. Interestingly, the authors that receive no feedback are most likely to leave a community. Furthermore, a structural analysis of the voter network reveals that evaluations polarize the community the most when positive and negative votes are equally split.
Introduction
The introduction asks whether community feedback guides authors toward behavior that benefits their communities, and reports instead that feedback can perpetuate undesirable behavior. It also highlights the asymmetry between negative and positive feedback and the polarization of voting networks.
Introduction
- The study addresses a gap in research on how ratings affect the behavior of the authors whose content is evaluated.It tests whether feedback regulates the quality and quantity of future contributions in ways that benefit the community.Source: Introduction · p. 1
- In four news communities, negative feedback preceded worse future posts and more frequent contributions, while positive feedback did not significantly improve post quality.The study analyzed about 140 million votes on 42 million comments; its findings run counter to the beneficial effects predicted by operant conditioning.Source: Introduction · pp. 1–2
- Voting networks were most polarized when up- and down-votes were equally split, and positive and negative votes tended to come from user communities.This structural analysis examined voter networks around popular posts.Source: Introduction · p. 2
- A crowdsourcing experiment validated the proportion of up-votes as a robust measure for aggregating community feedback.Source: Introduction · p. 2
- The authors introduced a propensity score matching framework to quantify community feedback's effects on post quality.Source: Introduction · p. 2
Further Related Work
This related-work section positions the paper as a framework for studying how votes affect the authors of evaluated content, rather than only content ratings or broader community phenomena. It also connects the findings to anti-social behavior, noting that negative feedback does not reduce undesired behavior.
Further Related Work
- Perceived post quality reflects both textual quality and community bias, which the paper disentangles through specifically designed experiments.The passage frames community bias as a factor in how the community evaluates a post.Source: Further Related Work · p. 2
- The reported observations cannot be attributed simply to flame wars because they unfold over a much longer timescale.Source: Further Related Work · p. 2
- Prior work examines review helpfulness, voting dynamics across platforms, and news-commenting phenomena, whereas this paper focuses on how votes affect evaluated authors’ behavior.It contributes a framework for analyzing vote effects on the author of the evaluated content.Source: Further Related Work · p. 2
- Negative feedback does not decrease undesired user behavior; the findings instead indicate that it attenuates that behavior.The discussion situates this result amid work on anti-social behavior, including review spamming, trolling, social deviance, and online harassment.Source: Further Related Work · p. 2
Measuring Community Feedback
Across four comment-based news communities, the study develops a vote-based measure of community feedback and validates it against people’s perceptions using crowdsourcing.
Dataset description
- The study analyzes comments and votes in four large news communities: CNN.com, Breitbart.com, IGN.com, and Allkpop.com.Timestamped activity from March 2012 to August 2013 covers 1.2 million threads, 42 million comments, 140 million votes, and 1.8 million users; analyses track users from joining onward. About 50% of posts received at least one vote and 10% received at least ten.Source: Dataset description · pp. 2–3
Measures of Community Feedback
- Posts were classified as positively evaluated in the community-specific upper quartile of up-vote proportions and negatively evaluated in the lower quartile.The measure is p = P/(P + N); posts needed at least ten votes for sufficient precision, and the upper quartile was often already 100% positive because proportions were skewed.Source: Measures of Community Feedback · pp. 3–4
- 0.92 was the highest R2: the proportion of up-votes best captured perceived feedback and was used as an intuitive, robust measure.In a crowdsourcing experiment, 66 workers rated 4,302 up-vote/down-vote pairs, each with at least nine independent evaluations.Source: Table 2 · p. 3, Measures of Community Feedback · p. 3
Post Quality
Negative feedback is followed by lower textual quality and worse community evaluations of authors’ future posts, while positive feedback has no significant, consistent effect on textual quality. This reverses the expected improvement from punishment or reward and suggests community perception also shapes the up-votes posts receive.
Textual vs. Community Effects
- The text-only model’s predictions correlate more with human quality labels (R2 = 0.25) than observed up-votes do (R2 = 0.12), supporting its use to separate textual quality from community perception.The model predicts up-vote proportion from text features; the difference between observed and predicted votes (p − q) measures community bias.Source: Textual vs. Community Effects · p. 4
- The study matches users on post quality and prior behavior, then compares the textual quality of their next three posts after positive versus negative evaluations.Matched posts at the evaluation event have textual quality within 10^-4, and users are also controlled for posting volume and past up-vote rates.Source: Textual vs. Community Effects · p. 5, Figure 3 · p. 5
How feedback affects a user’s post quality.
- Text quality significantly drops after negative evaluations (p < 0.05), while positive-feedback effects are inconsistent and not significant across communities.The reported negative-feedback effect is similar within and across threads (average r = 0.19 and 0.18, respectively).Source: How feedback affects a user’s post… · p. 5, Summary · p. 6, Figure 4 · p. 6
How feedback affects community perception.
- After negative evaluations, posts are perceived worse than text-only predictions, with a larger discrepancy than after positive evaluations (p < 10^-16, r > 0.03).The effect is stronger within threads (average r = 0.57) than across threads (average r = 0.13); on CNN, same-thread posts are evaluated 48 percentage points lower than their text predicts.Source: How feedback affects community… · pp. 5–6, Summary · p. 6, Figure 4 · p. 6
Summary.
- Authors’ predicted text quality falls after punishment, while rewarded authors’ quality remains unchanged, contrary to the expected quality improvements from feedback.The analysis compares subsequent posts with matched pre-evaluation posts of similar textual quality.Source: How feedback affects a user’s post… · p. 5, Summary · p. 6
- The detrimental effects of negative feedback are more noticeable than the beneficial effects of positive feedback, an asymmetry the authors relate to the negativity effect.The authors cite Kanouse and Hanson (1972) and Baumeister et al. (2001).Source: Summary · p. 6
User Activity
Negative evaluations lead authors to post more frequently than positive evaluations, while authors who receive no feedback slow down. Punished users are also more likely to stay than rewarded users, and withholding feedback may help discourage unwanted behavior.
User Activity
- 15% less frequently for no-feedback users, compared with 20% more for positively evaluated users and 30% more for negatively evaluated users; the negative-feedback effect exceeds the positive one (p < 10−15, r > 0.18).Propensity-score matching compared users with posts of the same textual quality and posting frequency before evaluation. The frequency effects were statistically significant and consistent across all four communities.Source: User Activity · pp. 6–7, Figure 5 · p. 7
- Punished users are more likely than rewarded users to stay and contribute more posts (χ2 > 6.8, p < 0.01), and both groups are less likely to leave than the average-user control group.The retention comparison could not control for users’ retention rates before evaluation.Source: User Activity · p. 7
- Because users who receive no feedback post less frequently, ignoring undesired behavior may be more effective than giving negative feedback to discourage it.Both positive and negative evaluations encourage users to post more frequently.Source: User Activity · p. 7
Voting Behavior
Negative evaluations are followed by more negative voting, while positive evaluations show no significant voting change. Voting behavior generally tracks the feedback users receive, though the most negatively evaluated users are an exception.
Voting Behavior
- Average users were most likely to leave the community, while rewarded users left sooner than punished users.Figure 6 defines retention as the fraction of users who posted at least x more posts, for a given x.Source: Figure 6 · p. 7
Tit-for-tat.
- 75%-upvote recipients gave up-votes only 67% of the time; very negatively evaluated users instead gave a higher proportion of up-votes than they received.Overall, the fraction of up-votes users receive is strongly linearly correlated with the fraction they give, suggesting tit-for-tat behavior.Source: Tit-for-tat · p. 7, Figure 7 · p. 7
- p < 10^-13 and r > 0.23: negatively evaluated users down-voted others more in the week after evaluation, while positively evaluated users showed no significant change.Figure 8 compares voting before and after evaluation.Source: Figure 8 · p. 8, Tit-for-tat · p. 8
- Punished users become more likely to evaluate fellow users negatively, potentially percolating the detrimental effects of negative feedback through the community.This adds a voting-behavior change to the previously observed posting-behavior change.Source: Tit-for-tat · p. 8
Organization of Voting Networks
Voting networks are most polarized when a post’s up- and down-votes are evenly split, and up-voters and down-voters generally vote less across camps than expected in randomly rewired networks. The structure of voter groups differs by community: patterns on CNN contrast with those on Breitbart, IGN, and Allkpop.
Experimental setup.
- Each voting network links users who voted on the same post if they voted on one another in the preceding 30 days, with edges signed by vote type.The analysis includes posts with at least ten votes and at least one up-vote and one down-vote.Source: Experimental setup · p. 8
When is the voting network most polarized?
- Evenly split votes produce the highest fraction of balanced triads, indicating the strongest coalitions and polarization across all four communities.A triad is balanced with either three positive edges or two negative edges and one positive edge; the fraction is normalized against shuffled edge signs.Source: When is the voting network most… · p. 8
- Cross-camp edges are fewest for evenly split votes and remain below randomly rewired levels across all four communities.The normalized cross-camp edge counts have negative Z-scores, suggesting up-voters and down-voters generally do not vote on each other.Source: When is the voting network most… · p. 9
Where does feedback come from?
- On CNN, the largest connected component shrinks as up-votes increase, while on Breitbart, IGN, and Allkpop it peaks when votes are almost all positive.This suggests positive-vote groups are more dispersed on CNN, while up-voters on the special-interest sites are likely to have voted on each other.Source: Where does feedback come from? · p. 9
Discussion
The discussion argues that peer feedback has community-wide behavioral effects, especially after negative evaluations, while identifying measurement limitations and directions for further study.
Discussion
- Across four diverse news communities, negative feedback—more salient than positive feedback—led authors to post more, produce lower-quality posts, and later evaluate peers negatively.The paper analyzes feedback at the user level and treats evaluations of content as implicit evaluations of its author.Source: Discussion · p. 9
- Observed behavior deviated from operant conditioning theory; online votes may be milder punishments than laboratory stimuli, and peer feedback may carry less authority than administrator feedback.The authors present these as potential explanations for the difference from theory’s predictions.Source: Discussion · p. 9
- The textual-quality regression model is a key limitation: despite validation against human labels, its fit could improve, enabling finer-grained analysis of feedback and post quality.Estimating text quality is described as a difficult machine-learning problem.Source: Discussion · p. 9
- Flame wars alone are unlikely to explain the findings because behavioral changes persist across threads and feedback providers change across an author’s posts.The scale and effects of such behavior remain open questions.Source: Discussion · p. 9
- Future research could compare feedback types and voting designs, study provider authority and user-specific responses, and analyze discussion language and context.The authors also propose surveys and controlled experiments to examine peer feedback’s role in norm enforcement, conformity, and potential deviant behavior.Source: Discussion · p. 9