Detecting Spin in Clinical Trials with Large Language Models

1 Introduction

• Spin distorts the presentation of study results, often benefiting authors, and can lead to redundant or misdirected follow-up research.The introduction notes that incorrect reporting persists even in RCTs, where reporting rules are clearly defined.Source: §1 · p. 1
• LLMs can classify primary and reported clinical-trial outcomes as similar or different, a core subtask in detecting outcome switching.Outcome switching is compromised reporting of primary and secondary outcomes in the results.Source: §1 · p. 1
• The study asks whether untrained LLMs can classify outcome pairs, how their generality and comprehension differ, and whether they can explain their decisions.Source: §1 · p. 1

2 Related Work

• Spin may mislead readers about treatment effectiveness and is reported in 47–71% of clinical-trial abstracts that fail to confirm their hypothesis, depending on field.Spin may be intentional or result from negligence.Source: §2 · p. 1
• Spin can involve misleading reporting, inappropriate interpretation, or unjustified extrapolation of results.Misleading reporting includes omitting, adding, or reordering outcomes; inappropriate interpretation includes unsupported conclusions, and extrapolation generalizes treatment effects to another population.Source: §2 · p. 1
• Automated spin-detection tools include extraction and bias-assessment systems; DeSpin breaks detection into extraction, detection, and classification, including outcome comparison [8, 12, 9].This paper addresses primary-versus-reported outcome comparison with LLMs, assuming the outcomes have already been extracted.Source: §2 · p. 1

3.1 Dataset

• 2,496 deduplicated outcome pairs formed the final test set, while 300 pairs were sampled for tuning and validation.The pairs label primary and reported outcomes as similar or different; similar-pair proportions were 31.0%, 21.5%, and 21.4% in tuning, validation, and test sets, respectively, versus 21.8% overall.Source: §3.1 · p. 2

3.2 Metrics

• The Youden index selected each model’s classification threshold, treating both classes equally for the imbalanced dataset.Accuracy, positive-class precision, recall, and F1 were reported, with F1 as the main metric.Source: §3.2 · p. 2

3.3 Baseline Models

• Baseline models spanned string, lexical, vector-based, ontology-based, and deep-learning similarity measures.The evaluated deep-learning baselines included SciBERT, BioBERT, and the all-MiniLM-L12 sentence transformer.Source: §3.3 · p. 2

3.4 Large Language Models

• The study evaluated instruction-tuned open-source LLMs, including transparent OLMo, general-purpose Mistral, biomedical BioMistral, and Llama.Local inference on a 16-GB GPU limited model size to 8B parameters; the models were selected for differing transparency, general, biomedical, or reasoning characteristics.Source: §3.4 · p. 2
• The probability-based method uses first-token probabilities to score Yes/No answers, avoiding limitations of extracting labels from generated text.Prompts expect finite-set answers, with each possible answer assigned a token; the first generated token supplies the probabilities.Source: §3.4 · pp. 2–3
• Prompts ranged from simple similarity questions to outcome-switching definitions and few-shot examples, and AUC on 200 validation instances guided prompt comparison.Complexity-3 prompts were selected as the starting point because their average AUC was comparable across prompts and their range across models was smallest.Source: §3.4 · p. 3, Figure 1 · p. 3, Table 1 · p. 3
• Misclassification review identified gaps in domain terminology and acronym knowledge, comprehension of longer sentences, and acceptance of partially reported outcomes.The prompt was iteratively clarified to require all components and details of the primary outcome in the reported outcome.Source: §3.4 · p. 3

3.5 Ensemble models

• Three-model hard-voting ensembles used the top validation-F1 models in each group; LLM’ voting combined OLMo, Mistral, and BioMistral with corrected prompts.The other ensembles were baseline voting and LLM voting with OLMo, BioMistral, and Llama using the initial clinical-reviewer prompt.Source: §3.5 · p. 3

4.1 Outcome Classification

• On 2,496 test instances, LLM’ voting reached F1 0.78 and accuracy 0.90, 7 percentage points above baseline voting.Source: §4.1 · p. 3, Table 2 · p. 4
• The sentence-transformer baseline achieved F1 0.67 and accuracy 0.82, while corrected-prompt OLMo’ led individual LLMs with F1 0.75 and accuracy 0.89.Source: §4.1 · p. 3

4.2 Explanation

• LLMs generated explanations after classification by continuing the conversation with the model prediction and asking, “Why? Explain your answer.”Source: §4.2 · p. 3
• The examples matched GDS scores to depression, but rejected pain alone against a primary outcome specifying pain severity and knee ROM.Source: §4.2 · pp. 3–4

• LLMs outperformed all baseline text-similarity models in F1, which the authors attribute to defining task context and the asymmetric outcome relationship.Source: §5 · p. 4
• Majority voting further improved results, with the models’ distinct mistakes indicating that they complemented one another.Source: §5 · p. 4
• Llama’s strict following of corrected instructions coincided with recall 0.33 and F1 0.49, while BioMistral and Mistral differed in prompt sensitivity.BioMistral performed better with minimal prompts, whereas Mistral benefited from more context.Source: §5 · p. 4
• Results trailed fine-tuned BERT by about 10 percentage points on F1; development used 10% of the dataset, and some generated explanations hallucinated.Thresholds tuned on a small random sample could be unstable across models; over 50% of explanations were judged valuable and approximately every tenth contained hallucinations.Source: §5 · p. 4

• Majority voting of three LLMs improved F1 to 0.78, while prompt-engineered OLMo achieved F1 0.75.Source: §6 · p. 4
• The approach uses only a few hundred instances without parameter fine-tuning and includes automatic threshold tuning, next-token-probability classification, and optional explanations.Source: §6 · p. 4
• Future work includes larger LLMs, other spin types and related tasks, whole-report analysis with longer-context models, and crosslingual evaluation.Source: §6 · p. 4