PaperlayerView exact revision on arXiv

Source-linked AI summary

Inferring Which Medical Treatments Work from Reports of Clinical Trials

Eric Lehman, Jay DeYoung, Regina Barzilay, Byron C. Wallace

arXiv:1904.01606v2cs.CL

TL;DR

Clinical-trial findings are published in natural-language articles, making the relevant evidence burdensome to review. This paper introduces evidence inference and a manually annotated RCT corpus, and finds that identifying prompt-relevant evidence remains difficult for current models.

  • Problem

    RCT findings are disseminated in natural-language articles, making it burdensome to consult all relevant evidence about whether a treatment works.

    Source: §1 · p. 1

  • Method

    The authors define evidence inference over a prompted intervention, comparator, and outcome, provide an annotated corpus, and evaluate baseline and neural models.

    Source: §1 · p. 2

  • Results

    Neural models perform best, but supervised attention gains are modest; access to oracle evidence spans yields a jump of over 20 points in F1 over the best end-to-end model.

    Source: §8 · p. 9

  • Takeaways & Limitations

    The authors identify conditional selection of relevant evidence spans as a key challenge and call for more sophisticated attention mechanisms.

    Source: §8 · p. 9

    Even with explicit pretraining and conditioning, the models do not learn to attend well to evidence relevant to each prompt.

    Source: §8 · p. 9

Abstract

from arXiv · show

How do we know if a particular medical treatment actually works? Ideally one would consult all available evidence from relevant clinical trials. Unfortunately, such results are primarily disseminated in natural language scientific articles, imposing substantial burden on those trying to make sense of them. In this paper, we present a new task and corpus for making this unstructured evidence actionable. The task entails inferring reported findings from a full-text article describing a randomized controlled trial (RCT) with respect to a given intervention, comparator, and outcome of interest, e.g., inferring if an article provides evidence supporting the use of aspirin to reduce risk of stroke, as compared to placebo. We present a new corpus for this task comprising 10,000+ prompts coupled with full-text articles describing RCTs. Results using a suite of models --- ranging from heuristic (rule-based) approaches to attentive neural architectures --- demonstrate the difficulty of the task, which we believe largely owes to the lengthy, technical input texts. To facilitate further work on this important, challenging problem we make the corpus, documentation, a website and leaderboard, and code for baselines and evaluation available at http://evidence-inference.ebm-nlp.com/.

1 Introduction

The paper frames evidence inference as identifying what a full-text RCT report says about a specified intervention, comparator, and outcome. It introduces a manually annotated corpus and finds that selecting relevant evidence from long articles remains a central modeling challenge.

1 Introduction

  • Trial reports are published at a rate exceeding 100 per day, making it time-consuming and often practically impossible to review all relevant literature.Evidence is largely disseminated in unstructured scientific manuscripts.Source: §1 · p. 1
  • Automating evidence synthesis could mitigate costs and help keep treatment recommendations up to date as evidence is published.Organizations including Cochrane and evidence-based practice centers manually synthesize findings but struggle to keep up with the literature.Source: §1 · p. 1
  • The task asks whether an article reports a significant increase, decrease, or no significant difference for a prompted intervention–comparator–outcome relationship.The model receives the prompt and full-text article and must provide supporting textual evidence.Source: §1 · p. 1, Figure 1 · p. 1
  • In 67% of corpus articles, different prompts have different labels, so generally positive language can mislead models that ignore the specified outcome.An article may report no significant difference for one outcome but a significant increase for another.Source: §1 · p. 2
  • Attention models gain only modest predictive improvements, and even pretrained, prompt-conditioned attention struggles to consistently identify relevant evidence.The authors identify conditional evidence selection as a key challenge for lengthy trial reports.Source: §1 · p. 2
  • The authors contribute a publicly available corpus of 10,000+ prompts, answers, evidence spans, and full-text articles annotated by medical doctors, alongside baseline algorithms.They also formulate evidence inference as a practically important and technically challenging task.Source: §1 · p. 2

2 Annotation

Medical doctors generated, annotated, and verified prompts against full-text RCT articles, with supporting evidence spans and quality checks at each stage. Pilot feedback improved annotator agreement, and verification removed invalid prompts and responses.

2 Annotation

  • Relevant evidence appeared in the abstract for only 40.5% of prompts, and medical doctors were assigned to prompt generation, annotation, or verification.Annotation was performed on full texts because relevant clinical evidence is often available only there.Source: §2 · p. 2

2.1 Prompt Generation

  • Each prompt specified an answerable intervention–comparator–outcome question with an answer and supporting evidence snippet; articles yielded an average of 4.19 prompts.Prompts concerned whether the intervention was significantly more effective than the comparator for the specified outcome.Source: §2.1 · pp. 2–3
  • Verifiers accepted prompt creators’ answers in 95.9% of cases and rationales in 97.8% of cases.A RobotReviewer RCT classifier was used to screen articles, though some false positives remained.Source: §2.1 · p. 3

2.2 Prompt Annotation

  • Annotators classified whether the intervention significantly increased, decreased, or made no significant difference to the outcome, and marked supporting text.They could also mark a prompt invalid if it was not answerable from the article; annotation duplicated the prompt creator’s labels and rationales for quality control.Source: §2.2 · p. 3

2.3 Verification

  • Verification marked 3.8% of prompts, 6.7% of answers, and 7.1% of rationales invalid; invalid prompts and those with all answers or rationales rejected were removed.Source: §2.3 · p. 4

2.4 Task Refinement

  • After personalized feedback, agreement on the same 93 prompts rose from Krippendorff’s α=0.58 to α=0.84, then reached α=0.75 among annotators and 0.80 with the prompt generator on 113 additional prompts.Source: §2.4 · p. 4

3 Dataset Statistics

The corpus contains 10,137 annotated prompts across 2,419 articles, with multiple independent labels and rationales per prompt. The authors report high overall agreement and articles averaging 4.2k tokens.

3 Dataset Statistics

  • 10,137 annotated prompts cover 2,419 unique articles, with at least two independent label-and-rationale sets for each prompt.Source: §3 · p. 4
  • Overall agreement was Krippendorff’s α=0.88 when verifier labels were converted to the labels they endorsed, and α=0.86 without verifier annotations.Source: §3 · p. 4
  • Articles average 4.2k tokens, while intervention, outcome, and comparator strings average 5.1, 5.3, and 3.4 tokens, respectively.Source: §3 · p. 4

4 Models

The section progresses from simple classification baselines to models that identify evidence in trial articles. It explores using marked evidence spans to train attention and prompt-sensitive evidence selection.

4.1 Baselines

  • Baselines include a majority-class predictor, overlap-and-p-value heuristics, and bag-of-words logistic regression over article and intervention, comparator, and outcome inputs.Source: §4.1 · p. 4
  • Neural variants encode prompt elements as vectors and articles with a GRU; the simplest model concatenates article and prompt vectors for feedforward classification.Source: §4.1 · p. 5

4.2 Finding the Evidence

  • Evidence-aware systems either tag evidence before a separate predictor or jointly infer using token attention; evidence can be tagged at token or sentence level and conditioned on the prompt.Source: §4.2 · p. 5
  • The linear pipeline uses one logistic regressor to select evidence sentences and another to classify from selected sentences; a neural pipeline encodes sentences with a GRU.Source: §4.2 · p. 5
  • Conditional attention concatenates intervention, comparator, and outcome vectors to GRU hidden states, then scores tokens through a single hidden layer.Source: §4.2 · p. 6, Figure 4 · p. 6
  • Marked spans supervise attention with uniform-distribution or binary token targets; because evidence depends on the prompt, unconditioned attention cannot isolate relevant spans across prompts.Source: §4.2 · p. 6

5 Experimental Details

Model development used a validation set for iterative selection before final held-out test evaluation. The experimental details specify neural training settings, attention pretraining, and pipeline model training.

5.1 Development Setting

  • Final experiments were selected using the standardized validation set before held-out test evaluation; neural results are averages over five runs to mitigate random-initialization and fitting noise.Source: §5.1 · p. 6

5.2 Training Details and Hyperparameters

  • Neural models trained for up to 50 epochs with patience 10, using 32-unit GRU encoders and batch size 32; pretrained PubMed embeddings were not fine-tuned.Source: §5.2 · p. 6

5.3 Attention pretraining Details

  • Attention components were pretrained against marked evidence using token-wise binary cross-entropy, with the best validation token AUC initializing end-to-end inference; freezing attention parameters performed poorly.Source: §5.3 · p. 6

5.4 Pipeline Model Training Details

  • Pipeline sentence taggers were trained on labels derived from token annotations, and the document-level model then consumed only sentences tagged as relevant.Source: §5.4 · p. 7

6 Results

Pretrained conditional attention is the reported best-performing variant, but end-to-end models remain far behind oracle performance and poorly focus on evidence. Attention AUC also falls when training shifts from pretraining to the downstream task.

6.1 Main Task Results

  • Pretrained conditional attention is the best-performing model and current leaderboard leader.Table 4 reports leaderboard results alongside a standard attentive architecture for context.Source: §6.1 · p. 7, Table 4 · p. 7
  • Using only the prompt surprisingly outperforms a vanilla neural model, while ignoring the prompt severely degrades performance.Source: §6.1 · pp. 7–8
  • End-to-end models remain over 20 points behind the oracle variant that receives reference evidence spans.Table 6 reports results for models given reference evidence spans.Source: Table 6 · p. 8, §6.2 · p. 8

6.2 Analyzing Attention

  • Even pretrained models assign less than 14% of total attention mass to evidence tokens.Figure 5 presents validation-set evidence token masses after training.Source: Figure 5 · p. 8, §6.2 · p. 8
  • For supervised attention variants, evidence-token AUC peaks during pretraining and drops sharply when training switches to the downstream task.Figure 6 plots validation evidence-token AUC over training epochs.Source: §6.2 · p. 8, Figure 6 · p. 9

7 Related Work

The task intersects several NLP and biomedical extraction areas, but prior work had not inferred trial findings for specified interventions and outcomes. The paper identifies that inference as its distinctive focus.

7 Related Work

  • The task sits at the intersection of information extraction, natural language inference, evidence mining, and question answering.Source: §7 · p. 9
  • Prior work targeted clinical-trial variables in articles or abstracts, interventions and outcomes (Blake and Lucic, 2015; 2012), and casual biomedical statements (Mihaila et al., 2013).Modality classification in scientific literature had also been investigated.Source: §7 · p. 9
  • These prior efforts did not infer findings about the extracted interventions and outcomes, which is the focus here.Source: §7 · p. 9

8 Conclusions and Future Work

The baseline results establish that evidence inference is feasible but difficult, with relevant-span identification the key challenge. Oracle access produces a large F1 improvement, while proposed models still attend poorly to evidence.

8 Conclusions and Future Work

  • Simple baselines perform poorly, while modern neural architectures currently achieve the best results, establishing both feasibility and difficulty.Source: §8 · p. 9
  • Providing reference evidence spans yields an over-20-point F1 advantage over the best end-to-end model.Source: §8 · p. 9
  • The best proposed models place only about 10–15% of attention mass on tokens marked as relevant evidence.Source: §8 · p. 9
  • The authors identify conditional evidence-span localization as a future research direction because current models attend poorly even with pretraining and conditioning.Source: §8 · p. 9

A Description of Heuristics Baseline

The rule-based baseline first selects a likely evidence sentence, then interprets it using p-value and direction heuristics. The appendix also reports per-prompt and per-article annotation costs.

A Description of Heuristics Baseline

  • The heuristic baseline has two stages: find the sentence containing the answer, then interpret that sentence; its evidence-span variant uses only the interpretation stage.Source: A Description of Heuristics Baseline · p. 12
  • Sentence selection ranks sentences by occurrences of outcome, intervention, and comparator words, choosing the highest-scoring sentence.Source: A Description of Heuristics Baseline · p. 12
  • The baseline searches for p-value forms and uses thresholds to assign significance labels, then counts increase and decrease synonyms to infer direction, favoring no difference on ties.When intervention or comparator terms occur in the sentence, the method seeks the closest p-value; otherwise it sums points across p-values.Source: A Description of Heuristics Baseline · p. 12

B Annotation Costing Details

  • Prompt generation averaged 99 cents per prompt and about $4.14 per article; prompt annotation cost 58 cents and verification 27 cents per prompt.Source: B Annotation Costing Details · p. 12

C Additional Dataset Details

This appendix describes annotation-row metadata and XML preprocessing, then compares attention variants and criteria for selecting pretrained weights.

C Additional Dataset Details

  • Prompt-generator and annotator rows are distinguished by UserID, and rationale offsets are based on preprocessed XML and calculated through FuzzyWuzzy.Each row also records the verifier’s assessment of the answer and rationale.Source: C Additional Dataset Details · pp. 12–13

D Preprocessing Details

  • The XML preprocessing parses documents by section and subsection, removes HTML tags, and deletes residual paragraph tags from malformed documents.Source: D Preprocessing Details · p. 13

E Attention (and Attention Pretraining) Variants

  • Attention variants include independent tokenwise predictions, balanced evidence/non-evidence training samples, and an objective maximizing attention mass on evidence tokens.Tokenwise activations are normalized when forming the context vector; pretraining selects weights using evidence-token AUC.Source: E Attention (and Attention… · p. 13

F Validation Results and Variances

The validation appendix reports five-run means and ranges for model variants and describes the criteria used to select pretrained weights.

F Validation Results and Variances

Related papers

Loading 1904.01606v2…