Source-linked AI summary
UniBuc at SemEval-2024 Task 2: Tailored Prompting with Solar for Clinical NLI
Marius Micluta-Campeanu, Claudiu Creanga, Ana-Maria Bucur, Ana Sabina Uban, Liviu P. Dinu
TL;DR
Clinical-trial reports contain extensive information that is challenging to analyze manually. UniBuc tested SOLAR-based summarization and tailored prompting for clinical NLI; its reported findings identify shortcut reliance, especially under semantic-preserving changes, and its system placed 14th of 32 teams.
Problem
ClinicalTrials.gov hosts more than 480,000 trials, making manual analysis and information extraction challenging.
Source: §1 · p. 1Method
The approach summarized clinical-trial report sections and used prompting for inference, with prompt instructions and content order tuned for the task.
Source: §4.2 · p. 3, §6 · p. 5Results
Takeaways & Limitations
The authors conclude that benchmark metrics and semantic interventions reveal shortcut use that surface-level clinical NLI performance may not show.
Source: §7 · p. 6Summarization risks discarding essential information, while hypothesis-conditioned summaries sometimes included the hypothesis or contradictory phrases.
Source: §4.2 · p. 3
Abstract
from arXiv · showhide
This paper describes the approach of the UniBuc team in tackling the SemEval 2024 Task 2: Safe Biomedical Natural Language Inference for Clinical Trials. We used SOLAR Instruct, without any fine-tuning, while focusing on input manipulation and tailored prompting. By customizing prompts for individual CTR sections, in both zero-shot and few-shots settings, we managed to achieve a consistency score of 0.72, ranking 14th in the leaderboard. Our thorough error analysis revealed that our model has a tendency to take shortcuts and rely on simple heuristics, especially when dealing with semantic-preserving changes.
1 Introduction
The paper addresses the difficulty of analyzing extensive clinical-trial information and presents UniBuc’s prompting and input-manipulation approach for clinical-trial inference. The team placed 14th of 32 teams, while identifying numerical reasoning and rephrasing as major challenges.
1 Introduction
- ClinicalTrials.gov hosts more than 480,000 trials, making manual analysis and information extraction challenging; SemEval-2024 Task 2 tests LLM inference and robustness on clinical text.The task uses expert-annotated data with semantic interventions to assess safety and robustness.Source: §1 · p. 1
- The approach used SOLAR-Instruct with targeted summarization and prompts tailored to individual clinical-trial report sections in zero-shot and two-shot settings.The summarization strategy retained only the first sentence containing “trial.”Source: §1 · p. 1
- 14th of 32 teams was UniBuc’s leaderboard placement, despite challenges with numerical reasoning and rephrasing.The authors also report that causal models significantly outperformed masked language models on the task.Source: §1 · p. 1
2 Related Work
Prior clinical-trial research has examined contradiction detection, eligibility assessment, hallucination, and reasoning, while the previous NLI4CT task mostly used pretrained language models. UniBuc adds input preprocessing and reports that the general LLMs it tested performed as well as or better than the medical models it tested, while cautioning that the comparison was not exhaustive.
2 Related Work
- Prior clinical-trial work covered contradiction detection, LLM-based eligibility assessment, and healthcare hallucination and reasoning, while most prior NLI4CT systems used pretrained language models.A small minority of previous systems used ontologies and rule-based algorithms.Source: §2 · p. 1
- Unlike most language-model approaches, UniBuc preprocesses clinical-trial inputs, including through summarization, before model inference.The authors describe this as part of their attempt using both discriminative and generative language models.Source: §2 · pp. 1–2
- The tested general LLMs performed as well as or better than the medical models tested, but the authors caution that the comparison was not exhaustive.They used few medical LLMs and therefore draw no definitive conclusion about the comparison.Source: §2 · p. 2
- NLI4CT uses Clinical Trial Reports, whereas MedNLI uses clinical notes; clinical-trial inference also requires numerical reasoning because reports contain substantial quantitative information.The authors used SOLAR 10B, trained on mathematical-domain question-answer pairs to enhance mathematical capabilities.Source: §2 · p. 2
3 Data and Task Description
The task evaluates whether models can make faithful and consistent clinical-trial inference under semantic interventions. Its dataset contains expert-annotated entailment relations drawn from breast-cancer trial reports.
3 Data and Task Description
- The dataset comprises 1,000 breast-cancer clinical trial reports and 24,000 expert-annotated entailment relations, with four report sections: eligibility criteria, intervention, results, and adverse events.The reports were collected from ClinicalTrials.gov.Source: §3 · p. 2
- Each example pairs a premise from one or two trial reports with a statement and an entailment or contradiction label.Single instances use one trial; Comparison instances may use primary and secondary trials.Source: §3 · p. 2
- The test set contains 5,500 entailment relations, compared with 1,700 in training and 200 in development.The evaluation data includes semantic-altering and semantic-preserving interventions.Source: §3 · p. 2
- Faithfulness measures prediction adjustment after semantic-altering interventions, while Consistency measures label agreement across semantic-preserving interventions.The task also uses F1-score.Source: §3 · p. 2
4 Methodology
The methodology combines fine-tuned pretrained models with large language models and experiments with summarization and tailored prompting for clinical-trial inference. The SOLAR pipeline shortens report sections before prompting, trading information retention against faster repeated inference and prompt development.
4.1 Pre-trained Masked Language Models
- Pretrained-model experiments fine-tuned PubMedBERT, XLM-RoBERTa, and DeBERTa V3 Large in two stages, first freezing pretrained weights and then training the classifier with the model’s last layer.The first stage ran for four epochs at a learning rate of 10^-3; the second trained for one more epoch at 2 * 10^-4.Source: §4.1 · p. 2
- The team also tested a BioBERT sentence-transformer through a cross-encoder trained for NLI4CT sentence-pair classification, and used MedNLI-trained SciFive zero-shot.The BioBERT cross-encoder was trained on NLI4CT data for 20 epochs.Source: §4.1 · pp. 2–3
4.2 Large Language Models
- SOLAR-Instruct was selected for instruction tuning and enhanced mathematical capabilities; LLaMa-2 and Mistral were tested zero-shot with one shared prompt across report sections.The SOLAR approach instead summarized clinical-trial reports before inference.Source: §4.2 · p. 3
- The SOLAR pipeline summarized report sections to shorten context, then classified with two-shot prompts; preliminary tests found this outperformed using full Results sections.The authors did not run further full-report experiments because full-report inference took longer and summaries enabled more prompt refinement.Source: §4.2 · p. 3
- SOLAR section summaries yielded 0.63–0.72 F1-score in preliminary single-Results trials, compared with 0.55 F1-score without a summary.Summaries were capped at 350 tokens; shortened reports took roughly 5 seconds to evaluate versus up to 20–30 seconds for full reports.Source: §4.2 · p. 3
- Summarization risks discarding essential information, and hypothesis-conditioned summaries sometimes repeated the hypothesis or introduced contradictions.The authors avoided contextualized summaries because generating one per statement was costly when sections were evaluated against multiple statements.Source: §4.2 · p. 3
- For inference, the system kept the first sentence containing “trial,” used section- and example-specific prompts, and restricted outputs to “Yes” or “No.”The authors preferred this faster sentence-splitting heuristic to extracting relevant sentences from the hypothesis.Source: §4.2 · pp. 3–4
5 Results
UniBuc’s system scored 0.72 on Consistency (14th of 32 teams) and 0.83 on Faithfulness (10th). On the control set, it achieved an F1-score of 0.71.
5 Results
- 0.72 Consistency (14th of 32 teams) and 0.83 Faithfulness (10th of 32 teams) show stronger performance on semantic-altering than semantic-preserving interventions.Only one CTR section had Faithfulness below 0.79: eligibility comparisons, at 0.71.Source: Table 1 · p. 4, §5 · p. 4
- 0.65 and 0.72 F1 scores for zero-shot LLaMa-2 and Mistral were accompanied by low Faithfulness and Consistency performance.The paper reports no significant performance improvement from clinical-text pretraining and says generic instruction-tuned models performed better than biomedical-pretrained discriminative models.Source: §5 · p. 4
- 0.71 F1-score on the control set included 0.79 F1 for adverse-event comparisons and 0.62 F1 for intervention comparisons.The team ranked 16th of 32 on the control set.Source: §5 · p. 4
6 Error Analysis and Discussion
The discussion examines SOLAR’s intervention-specific errors and reports how prompting, summarization, and model-size choices shaped the approach. The analysis identifies brittle numerical and wording-related behavior alongside trade-offs in few-shot prompting and summarization.
Error Analysis and Discussion
- Definition interventions append a sentence to the statement; extracting the relevant trial sentence separately was possible but added runtime, so the team used a heuristic.Source: §6 · p. 4
- Changing the trial’s cycle length produced entailment predictions even for nonsensical values, an issue the authors observed only with two-shot prompting and associated with long contexts.For a trial stating a 21-day cycle, changing the statement’s number yielded contradiction for most tested values, whereas changing the trial’s number yielded entailment.Source: §6 · p. 5
- 0.83 Faithfulness, 0.78 Consistency, 0.89 F1, 1.0 Recall, and 0.80 Precision for contradictions contrast with 0.70 Consistency, 0.69 F1, 0.72 Recall, and 0.66 Precision for paraphrases.The authors attribute perfect recall to over-sensitivity to wording changes, which can make paraphrases appear conflicting.Source: §6 · p. 5
- 0.67 Consistency, 0.54 F1-score, 0.63 Recall, and 0.47 Precision for numerical paraphrasing contrast with 0.80 Faithfulness, 0.85 Consistency, 0.91 F1, 1.0 Recall, and 0.83 Precision for numerical contradictions.The model struggles with equivalent quantities expressed in different units, and lowercase domain acronyms can change its prediction.Source: §6 · pp. 4–5
- Two-shot prompting improved development-set F1 on average, but test-set Faithfulness and Contradiction declined while paraphrasing scores improved.Six-shot prompting performed poorly in preliminary development tests and increased runtime; the authors suggest complexity may explain the decline.Source: §6 · p. 5
- Prompt order mattered: putting the hypothesis before the CTR summary cost 4 F1 percentage points on development, while instructions at the prompt’s start performed best.Repeating the instruction before and after the CTR-hypothesis pair was as effective as placing it before the CTR text.Source: §6 · p. 5
- Larger models performed better, while smaller models helped prototype prompts; a hybrid using a smaller summarizer had performance comparable to an all-larger-model pipeline.Source: §6 · p. 5
- Summary prompts had no substantial effect on final results, but summaries needed to preserve source meaning and avoid depending on the hypothesis.For a Results section with multiple hypotheses, conditioning a shared summary on the first hypothesis could alter the CTR’s meaning.Source: §6 · pp. 5–6
7 Conclusions and Future Work
The paper presents an approach to understanding LLM behavior in clinical contexts and finds that models can rely on shortcuts, especially under semantic-preserving changes. It proposes specialized models as a possible way to address numerical reasoning weaknesses.
7 Conclusions and Future Work
- The team presents its SemEval 2024 Task 2 approach to studying LLM behavior in clinical contexts.Source: §7 · p. 6
- Models show surface-level clinical NLI capability but tend to use shortcuts and simple heuristics, especially with semantic-preserving changes.The authors plan to investigate further methods for evaluating LLM reliability.Source: §7 · p. 6
- For complex mathematical hypotheses, the authors propose offloading numerical reasoning to a specialized model such as xVal.They describe representing numbers as digits, replacing them with a [NUM] token, and scaling that token according to the original value.Source: §7 · p. 6
A.1 Summary mismatch example
The example shows how a summary of trial results can omit a detail needed to assess a particular hypothesis. The authors identify short-section summarization as a possible mitigation.
A.1 Summary mismatch example
- The trial measured change in total body lean mass from baseline to four months using DEXA.Source: §A.1 · p. 7
- The 1 mg arm had a median change of 1.55 kg among 32 participants, while the 3 mg arm had a median of 0.98 kg among 34 participants.Source: §A.1 · pp. 7–8
- A summary conditioned on a hypothesis about all 1 mg patients gaining lean mass over 10 years omitted the “over 10 kilos” detail needed by another hypothesis.The authors suggest refraining from summarizing short sections as a possible mitigation.Source: §A.1 · p. 8
B Prompt templates
The appendix presents initial prompts for summarization and NLI evaluation. The evaluation prompt requests a Yes-or-No inference judgment from a report and hypothesis.
B Prompt templates
- The team began with one prompt template across sections and summaries, then examined section-level F1 scores when results stopped improving.Due to time constraints, summaries were created for only the first 50 training examples.Source: B Prompt templates · p. 8
- The initial summarization prompt asked for a brief report summary using abbreviations, capped at 350 words.Source: B Prompt templates · p. 8
- The initial evaluation prompt supplied a clinical trial report and hypothesis, then requested only a Yes-or-No inference judgment.The same prompt was used in the LLaMa-2 and Mistral experiments.Source: B Prompt templates · p. 8
C Infrastructure
The appendix reports hardware, summarization and evaluation runtimes, and the scope of the initial-results and prompt-template tables. It notes that two-shot prompting increases total evaluation time.
C Infrastructure
- The system used 16 GB RAM and an NVIDIA GTX 1060 MQ GPU with 6 GB VRAM; development-set summarization required 100 summaries for 200 samples.The test set required summaries for 251 examples comprising different CTR sections.Source: C Infrastructure · p. 8
- The appendix identifies initial results for the first 50 training examples and for the development set, alongside section-specific evaluation and summarization templates.Table 4 covers evaluation templates, and Table 5 covers summarization templates by CTR section.Source: Table 2 · p. 8, Table 3 · p. 8, Table 4 · p. 9
- Evaluation took roughly 9 hours on the test set and about 12 hours with two-shot prompting, while summary generation was a one-time cost.A summary took at least 30 seconds per sample; inference took approximately 5 seconds per example.Source: C Infrastructure · p. 8