People on Drugs: Credibility of User Statements in Health Communities

1. INTRODUCTION

• Credibility concerns constrain the use of online health content, even though these communities are a potential source of information for patients and physicians.The paper focuses on rare or unknown drug side-effects, where large-scale non-expert data may complement expert knowledge and misinformation can have hazardous consequences.Source: §1 · p. 1
• The model jointly assesses statement credibility, user trustworthiness, and post-language objectivity, using linguistic and user features with expert-labeled side-effect statements.A probabilistic graphical model supports joint inference; the method uses an EM framework with MCMC sampling in the E-step and Trust Region Newton feature-weight estimation in the M-step.Source: §1 · pp. 1–2
• 82% overall accuracy in identifying drug side-effects, improving 13% over an SVM with the same features and 4% over a stronger SVM using distant supervision.The evaluation also considers discovering rare side-effects and identifying trustworthy community users.Source: §1 · p. 2
• The paper applies this approach to extracting medical-drug side-effects from online health forums and evaluates two practical use cases.The proposed model captures interactions among trustworthiness, objectivity, and credibility.Source: §1 · p. 2

2. OVERVIEW OF THE MODEL

• The Markov Random Field assigns binary variables for statement credibility, post objectivity, and user trustworthiness, with observed user and post features and expert-derived labels for some statements.The model seeks credibility labels for unobserved statements by leveraging the variables' mutual influence.Source: §2 · p. 2
• The graph connects users to their posts, statements to posts from which they are extracted, and users to statements appearing in their posts.Shared nodes allow information to pass between cliques, including cross-talk among users and statements.Source: §2 · p. 2
• Inference also yields likelihoods that can rank statements by credibility, users by trustworthiness, and posts by objectivity.Figure 1 summarizes the interactions among the three modeled qualities.Source: §2 · p. 2, Figure 1 · p. 3

3.1 Linguistic Features

• Stylistic cues such as uncertainty markers, hypotheticals, and questions distinguish uncertain posts from more definite, inferential language.The feature set uses relative frequencies of stylistic word types in posts and aggregates post vectors by user.Source: §3.1 · p. 3
• Affective features encode post attitudes and emotions using WordNet-Affect attributes, mapped to 285 categories without word-sense disambiguation.The method takes each word's most common sense and constructs an affective feature vector for each post.Source: §3.1 · p. 3, Table 2 · p. 4
• In the preliminary analysis, confidence, pride, affection, positivity, and inferential statements about definite entities correlate with user helpfulness, unlike negative affect and hypothetical or uncertain language.Helpfulness is proxied by received thanks; the learned preliminary weights are not used to train the final model, which instead uses partially provided expert information.Source: §3.1 · p. 4

3.2 User Features

• User features include demographics, engagement and interaction measures, and the first three moments of each user's post-length distribution in sentences and words.The paper treats these characteristics as intuitive correlates of user authority and post focus.Source: §3.2 · p. 4

• Expert-labeled true/false side-effect statements supervise inference of labels for the remaining unlabeled statements.Source: §4 · p. 4
• Each MRF clique links a statement instance, its containing post, and author; its factors use stylistic, affective, and user features.Source: §4 · p. 4
• The model estimates statement labels conditionally on posts and users, using a CRF formulation rather than modeling the full joint distribution.Source: §4 · p. 4
• EM alternates Gibbs sampling of unlabeled labels with weight estimation; the M-step uses L2-regularized Trust Region Newton via LibLinear.Source: §4 · pp. 4–5
• An explicit user score is the fraction of that user's statements judged true in the previous EM iteration.Source: §4 · p. 5
• Inference iterates trustworthiness estimation, unlabeled-statement sampling, and feature-weight updates; statement labels depend on post objectivity and users' trustworthiness.Source: §4 · p. 5

5.1 Data

• The study uses 2.8 million posts from 15,000 healthboards.com users and selects six widely used drug families, using Mayo Clinic information as expert ground truth.The expert portal categorizes side-effects as common, less common, or rare.Source: §5.1 · pp. 5–6

5.2 Baselines

• The model is compared with frequency and SVM baselines; its semi-supervised CRF shares information across instance cliques, and predictions for each side-effect are consolidated by majority vote.Evaluation averages results over 200 random 80/20 train-test splits for each drug family.Source: §5.2 · p. 6, §5.3 · p. 6

5.4 Results and Discussions

• The CRF model improves average accuracy by 6% over the SVM L1 baseline and 4% over the SVM L2 baseline through coupling users, posts, and statements.The reported feedback loop allows information to flow between cliques.Source: §5.4 · p. 7
• Overall, the approach increases sensitivity by 5% and specificity by 3% over the SVM L2 baseline.Figure 3 compares models on specificity and sensitivity.Source: §5.4 · p. 7, Figure 3 · p. 7
• In Setting I, specificity gains over SVM L2 peak at 8.33% for Alprazolam and 4.6% for Levothyroxine; sensitivity gains reach 7.89%, 6.5%, and 6.33% for Metronidazole, Metformin, and Omeprazole.The authors report that the model performs best in these three latter drug categories.Source: §5.4 · p. 7
• In Setting II, accuracy improves by 2–3% for Metformin, Levothyroxine, and Omeprazole, but Alprazolam specificity drops by 8.42%.The authors attribute the Alprazolam drop to more false positives when less common effects become positive ground-truth.Source: §5.4 · p. 7, Table 7 · p. 7
• Affective features are most informative, followed by document-length statistics, while combining all features yields a 70% F-score for Alprazolam.Individual feature-set F-scores for Alprazolam range from 51% to 55%.Source: §5.4 · p. 7
• The strongest user feature is the ratio of replies to questions posted, followed by gender, post count, and thanks received.77.69% of active contributors in the forum are female.Source: §5.4 · p. 7

6.1 Discovering Rare Side Effects

• For Alprazolam and Levothyroxine, the study uses Mayo Clinic as its expert knowledge base and evaluates reported side-effects beyond its listings.The experiment covers side-effects flagged as very rare as well as those absent from the knowledge base.Source: §6.1 · p. 8
• The model ranks out-of-KB side-effects; two experts judge the top ten and ten randomly selected items, and ranking quality is measured with NDCG.Judges read discussion threads and other threads involving the reporting users; majority vote sets the label.Source: §6.1 · p. 8
• CRF ranked out-of-KB side-effects perfectly for both drugs; frequency ranking was poor for Alprazolam but performed quite well for Levothyroxine.The comparison uses expert-annotator judgments of the ranked side-effects.Source: Table 8 · p. 8, §6.1 · p. 8

6.2 Following Trustworthy Users

• After moderators and facilitators are removed, judges rate top-ranked users against top-thanked contributors; CRF outperforms the baseline for both drugs.The baseline performs poorly for Levothyroxine, while judges show good agreement for Alprazolam and very good agreement for Levothyroxine.Source: §6.2 · p. 8, Table 9 · p. 8

Information Extraction

• Information-extraction methods extract statements but do not judge their credibility; prior biomedical extraction largely targets scientific publications, with social media playing a minor role.The cited work covers biomedical facts and events, drug-experience reports, and health knowledge bases.Source: §7 · pp. 8–9

Truth Finding

• Prior truth-finding methods assess Web claims or medical claims, but are not designed to jointly model statement credibility, user trust, and language in discussion forums.The paper notes that none of these prior works considered online discussion forums.Source: §7 · p. 9

Language Analysis for Social Media

• Prior language-analysis research covers reviews, writing style, biased and subjective language, and its benefits for information extraction, but not online health-forum discussions.The cited work includes,,,, and.Source: §7 · p. 9

Trust and Reputation Management

• Most trust-management work infers authority from explicit user relationships; the closest content-aware model uses generic word-level evidence, unlike this paper's rich language features and online users.The paper presents its setting as requiring a more sophisticated model.Source: §7 · p. 9

8. CONCLUSION

• The probabilistic graphical model jointly learns user trustworthiness, statement credibility, and language use, leveraging user interactions and stylistic and affective language features.The approach is used to extract drug side-effects and filter false information in online communities.Source: §8 · p. 9
• Across experiments against expert knowledge and a user study, the model clearly outperforms all baselines.The experiments address rare side-effect discovery and identifying trustworthy users to follow.Source: §8 · p. 9
• The authors see the model as a possible asset for in-depth analysis of conditions associated with observed side-effects, including age, gender, social group, lifestyle, and other medication.Source: §8 · p. 9
• Simple information-extraction machinery misses some paraphrases, such as “nightmares” and “unusual dream” for Xanax, reducing recall.The authors suggest a more powerful extraction approach could further improve quality.Source: §8 · p. 9