PaperlayerView exact revision on arXiv

Source-linked AI summary

Balanced and Explainable Social Media Analysis for Public Health with Large Language Models

Yan Jiang, Ruihong Qiu, Yi Zhang, Peng-Fei Zhang

arXiv:2309.05951v1cs.CL

TL;DR

Social-media public-health analysis faces imbalanced datasets, costly LLM training, and BERT’s limited input size. ALEX combines augmentation and weighted-loss BERT fine-tuning with LLM inference to explain and correct predictions, ranking first in two SMM4H 2023 tasks and achieving a top result in a third.

  • Problem

    Public-health social-media classification is challenged by costly LLM training, BERT’s limited input size, imbalanced data, and unequal importance across classes.

    Source: §1 · p. 2

  • Method

    ALEX uses augmentation and weighted-loss fine-tuning for balance, then prompts an LLM with BERT predictions and original text to explain and correct labels.

    Source: §1 · p. 2

  • Results

    ALEX ranked first in SMM4H 2023 Tasks 2 and 4 and achieved a top result in Task 1.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework applies LLM inference to explain and correct BERT predictions without training the LLM.

    Source: §1 · p. 2

    For Task 1, LLM correction reduced performance after balanced training had brought Label 1 F1 to about 95, so the authors used balanced training alone.

    Source: §5.3 · p. 9

Abstract

from arXiv · show

As social media becomes increasingly popular, more and more public health activities emerge, which is worth noting for pandemic monitoring and government decision-making. Current techniques for public health analysis involve popular models such as BERT and large language models (LLMs). Although recent progress in LLMs has shown a strong ability to comprehend knowledge by being fine-tuned on specific domain datasets, the costs of training an in-domain LLM for every specific public health task are especially expensive. Furthermore, such kinds of in-domain datasets from social media are generally highly imbalanced, which will hinder the efficiency of LLMs tuning. To tackle these challenges, the data imbalance issue can be overcome by sophisticated data augmentation methods for social media datasets. In addition, the ability of the LLMs can be effectively utilised by prompting the model properly. In light of the above discussion, in this paper, a novel ALEX framework is proposed for social media analysis on public health. Specifically, an augmentation pipeline is developed to resolve the data imbalance issue. Furthermore, an LLMs explanation mechanism is proposed by prompting an LLM with the predicted results from BERT models. Extensive experiments conducted on three tasks at the Social Media Mining for Health 2023 (SMM4H) competition with the first ranking in two tasks demonstrate the superior performance of the proposed ALEX method. Our code has been released in https://github.com/YanJiangJerry/ALEX.

1 Introduction

The paper frames social-media public-health analysis as valuable but constrained by LLM training costs, BERT input limits, imbalanced data, and unequal class importance. ALEX combines augmentation and weighted-loss fine-tuning with LLM-based explanation and correction; it ranks first in two SMM4H 2023 tasks and achieves a top result in another.

1 Introduction

  • Social-media mining may inform public-health analysis, including vaccine acceptance estimates and government policy.The paper describes health-related datasets as abundant but their analytical value as largely untapped.Source: §1 · p. 1
  • Social-media public-health classification faces costly LLM training, limited BERT input length, imbalanced data, and differences in the importance of classes.The paper notes that imbalance may lead to biased predictions.Source: §1 · p. 2
  • The proposed ALEX framework combines data augmentation and weighted-loss fine-tuning with LLM inference to explain and correct BERT predictions.The LLM is prompted with the original text and BERT’s predicted label to check whether the text contains supporting evidence.Source: §1 · p. 2
  • ALEX ranks first in SMM4H 2023 Tasks 2 and 4 and achieves a top result in Task 1.All three tasks are text classification on public-health social-media datasets.Source: §1 · p. 2

2 Related Work

Related work reviews advances in text classification through pretrained, adapted, and architecturally modified models. It also describes LLMs’ growth and their potential use in health analysis.

2.1 Text Classification

  • Text classification research has used pretrained models such as BERT and GPT, with variants and architectural combinations explored for performance and resource efficiency.The cited examples include RoBERTa, ALBERT, DistilBERT, BERT-CNN, BERT-RCNN, and BERT-BiLSTM.Source: §2.1 · pp. 2–3
  • Domain-adapted models such as BERTweet, TwHIN-BERT, CamemBERT, and FlauBERT have achieved significant text-classification improvements in specific domains.BERTweet is described as the first open-source pretrained language model designed for English Tweets.Source: §2.1 · p. 3

2.2 Large Language Models

  • LLMs can be used for inference without gradient updates, and researchers have suggested their potential for disease surveillance using social-media data.The review describes GPT-1, GPT-2, GPT-3, GPT-3.5, and GPT-4 developments.Source: §2.2 · p. 3

3 Preliminaries: Task Definition

The paper formulates the three public-health social-media tasks as classification of posts into their corresponding labels.

3 Preliminaries: Task Definition

  • The task input is a set of social-media post texts with corresponding labels, and the objective is to classify the texts effectively.The setup uses x1 through xn for posts and y1 through yn for labels.Source: §3 · p. 3
  • Tasks 1 and 4 are binary classification, while Task 2 is three-class classification.These are the classification settings used across the paper’s three tasks.Source: §3 · p. 3
  • The paper’s notation distinguishes bold lowercase vectors, lowercase scalars or strings, and uppercase functions.This convention applies to the equations that follow.Source: §3 · p. 3

4 Methodology

ALEX balances social-media training data, fine-tunes BERT with a weighted loss, then prompts an LLM to examine BERT predictions and help correct labels.

4 Methodology

  • ALEX combines balanced training, weighted-loss BERT fine-tuning, and LLM-based label explanation and correction.The LLM receives the original post and BERT prediction in a prompt with instructions and examples.Source: §4 · p. 3, Figure 1 · p. 4, §4.4 · p. 5

4.1 Augmentation and Resampling

  • TextAttack augmentation followed by over- and undersampling makes the class counts equal.TextAttack is selected to enrich the minority class using adversarial attacks.Source: §4.1 · p. 4

4.2 Social Media Text Encoding

  • BERT encodes the balanced texts; its [CLS] token embedding feeds a classifier whose maximum-probability class is the predicted label.The [CLS] token is used to predict label probabilities.Source: §4.2 · p. 4

4.3 Weighted Loss

  • A weighted binary cross-entropy loss lets task objectives determine which labels receive greater training emphasis.The loss weight can be manually set according to the importance of a label for the task.Source: §4.3 · p. 5

4.4 LLM Explanation and Corrections at Inference Time

  • At inference, an LLM assesses BERT’s label against the post, retaining labels it supports and flagging unsupported or contradicted labels for correction.Flagged labels can be manually reviewed or directly changed; for three-class tasks, the direct-change option uses the majority label.Source: §4.4 · pp. 5–6

5 Experiments

The experiments evaluate three SMM4H 2023 public-health social-media tasks using validation data, classification metrics, and transformer baselines. The implementation combines augmentation and weighted-loss fine-tuning with task-specific settings and GPT-3.5 correction.

Datasets.

  • The study uses Task 1 English-language tweets, Task 2 therapy-sentiment tweets, and Task 4 Reddit social-anxiety posts, evaluating only the validation set.Task 1 includes 600,000 tweets; the competition provides labels for validation rather than testing.Source: Datasets · p. 6, Table 1 · p. 7

Metrics.

  • Task 1 and Task 4 are evaluated by Label 1 F1, while Task 2 uses micro-averaged F1, equivalent to accuracy.The baseline also reports macro-average F1 to show the effect of imbalance.Source: Metrics · pp. 6–7

Implementation.

  • The training pipeline augments minority-class samples with TextAttack, resamples labels to balance the training set, and fine-tunes with a tunable loss weight.BERT-related inputs are head-truncated to 512 tokens for large models and 128 for base models; XLNet inputs are not truncated.Source: Implementation · p. 7
  • The baselines include BERT, RoBERTa, XLNet, BERTweet, and CT-BERT, while GPT-3.5 explains and corrects selected BERT predictions.GPT-3.5 correction converts specified false predictions to task-dependent labels; the evaluated ALEX variants use BERTweet-Large for Tasks 2 and 4 and CT-BERT (v2) for Task 1.Source: Implementation · pp. 7–8

5.2 Overall Performance

ALEX-Large outperforms the other methods across all three tasks. Baseline performance varies by task and class, with different models leading on different comparisons.

5.2 Overall Performance

  • XLNet-Large scores zero on Task 1 Label 1 F1 and underperforms XLNet-Base there, while XLNet shows notable accuracy and macro-F1 on Tasks 2 and 4.RoBERTa performs better than BERT on Tasks 1 and 4, but RoBERTa-Base underperforms BERT on Task 2.Source: §5.2 · pp. 8–9
  • BERTweet-Large generally has the strongest Label 1 F1 in Tasks 2 and 4, while CT-BERT (v2) has the highest Label 1 F1 among baselines in Task 1.In Tasks 1 and 4, Label 1 is the minority class; in Task 2, it is the majority class.Source: §5.2 · p. 9
  • ALEX-Large outperforms all other methods in all three tasks by integrating balanced training and LLM correction.The comparison is on the validation set.Source: §5.2 · p. 9

5.3 Ablation Study

The ablation study reports gains from balanced training across all tasks and additional GPT-3.5 correction gains on Tasks 2 and 4. On Task 1, correction lowers performance relative to balanced training alone.

5.3 Ablation Study

  • Balanced training improves results over imbalanced-training baselines across all tasks.The study presents this comparison in the ALEX ablation study.Source: §5.3 · p. 9, Table 3 · p. 9
  • GPT-3.5 explanation and correction raises Task 2 and Task 4 Label 1 F1 from about 86 to above 88 versus balanced training alone.For Task 2, GPT-3.5 verifies Label 0 predictions and corrects them to majority-class Label 1; it is constrained to selected predictions rather than directly classifying text.Source: §5.3 · p. 9
  • On Task 1, GPT-3.5 correction decreases performance versus balanced training alone after Label 1 F1 reaches about 95.The authors retain only balanced training for Task 1 because GPT-3.5 struggles to identify the remaining false positives.Source: §5.3 · p. 9

5.4 Strategies for Balanced Training

Balanced training combines augmentation with weighted-loss fine-tuning, outperforming the individual strategies across Tasks 1, 2, and 4. Its embeddings separate labels more clearly for Tasks 2 and 4, while Task 1 shows less separation benefit over a strong baseline.

5.4 Strategies for Balanced Training

  • “Focal” means focal-loss fine-tuning, “Weighted” means weighted-loss fine-tuning, “Aug” means augmentation, and “Balanced” combines augmentation with weighted loss.For Task 2, the class weights are 2:1:2; Tasks 1 and 4 use a loss weight of 2.Source: §5.4 · pp. 9–10
  • Combining augmentation and weighted-loss fine-tuning produces superior performance across all three tasks, with visible improvements for Tasks 2 and 4.Focal loss without augmentation decreases performance on all three tasks; weighted loss alone may improve Task 1 but does not clearly improve Tasks 2 and 4.Source: §5.4 · p. 10, Table 4 · p. 10

Visualisation of Embeddings.

  • For Tasks 2 and 4, balanced training separates labels more clearly in the [CLS] embedding visualization than the baseline.For Task 1, the separation benefit is less significant because the baseline already achieves high accuracy.Source: Visualisation of Embeddings · p. 10, Figure 3 · p. 10
  • The authors identify balanced training as effective for public-health text-classification tasks with low accuracy or low interclass distance.Source: Visualisation of Embeddings · p. 10

5.5 Effectiveness of LLMs Explanation and Correction

The section examines how prompts affect LLM-based explanation and correction. Relevant prompts improve Task 1 and Task 4 metrics by around two percent, while irrelevant prompts can reduce performance below baseline.

Case Study for Prompts.

  • The experiment compares a relevant prompt that guides classification correction with an irrelevant prompt that misdirects the LLM toward treating the task as regression.Source: Case Study for Prompts · p. 11, §5.5 · p. 11
  • An appropriate prompt improves F1 score and accuracy by around two percent in Tasks 1 and 4, whereas an irrelevant prompt performs worse than baseline.Even with a relevant prompt, Task 1 performance remains below its previous results.Source: Case Study for Prompts · p. 11, Table 5 · p. 11

A Study for LLMs Hallucinations.

The authors report that GPT-3.5 can produce incorrect label explanations and may carry information from earlier inputs into later judgments. Restricting which predictions it verifies is presented as a way to mitigate hallucination effects, with task-specific gains.

A Study for LLMs Hallucinations.

  • In Task 1, GPT-3.5 may successfully find COVID-related context even when the label is correct, yet still produce balderdash and falsely assert that 1 = 1 is False.Source: A Study for LLMs Hallucinations · p. 11
  • The authors suggest repeated COVID information in prior context may carry over to subsequent inputs and contribute to hallucinations.They relate this possibility to prior findings on biases in retaining knowledge from previous inputs.Source: A Study for LLMs Hallucinations · p. 11

Strategies for Preventing LLMs Hallucinations.

6 Conclusion

The paper concludes that ALEX combines balanced training with LLM-based explanation and correction for public-health social-media analysis. It reports strong SMM4H 2023 results, while an example also shows that an LLM explanation can be wrong.

6 Conclusion

  • Figure 4 illustrates an incorrect LLM explanation for its second input; green marks correct responses and red marks incorrect ones.Source: Figure 4 · p. 12
  • ALEX combines balanced training for imbalance with LLM explanation and correction, and the authors report remarkable SMM4H 2023 competition results.Source: §6 · p. 12

Related papers

Loading 2309.05951v1…