PaperlayerView exact revision on arXiv

Source-linked AI summary

In-BoXBART: Get Instructions into Biomedical Multi-Task Learning

Mihir Parmar, Swaroop Mishra, Mirali Purohit, Man Luo, M. Hassan Murad, Chitta Baral

arXiv:2204.07600v1cs.CL

TL;DR

Task-specific models are computationally expensive and time-consuming, and generalization across biomedical NLP tasks had not been systematically studied. The paper creates BoX, a collection of 32 biomedical instruction tasks, and trains the unified In-BoXBART model on this instruction-based meta-dataset. In-BoXBART outperforms the single-task baseline by ∼3% and the multi-task baseline without instructions by ∼18%.

  • Problem

    Task-specific models are computationally expensive and time-consuming, and generalization across biomedical NLP tasks had not been systematically studied.

    Source: §1 · p. 1

  • Method

    The paper creates BoX, a collection of 32 biomedical instruction tasks, and trains the unified In-BoXBART model on this instruction-based meta-dataset.

    Source: §1 · p. 2

  • Results

    In-BoXBART outperforms the single-task baseline by ∼3% and the multi-task baseline without instructions by ∼18%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors identify instruction tuning as a potential approach for biomedical settings where obtaining annotated data is difficult.

    Source: §1 · p. 2

    For multi-class tasks, including one example per class can exceed language-model input limits, while using fewer labels or omitting examples may cause label bias or performance degradation.

    Source: §6 · p. 9

Abstract

from arXiv · show

Single-task models have proven pivotal in solving specific tasks; however, they have limitations in real-world applications where multi-tasking is necessary and domain shifts are exhibited. Recently, instructional prompts have shown significant improvement towards multi-task generalization; however, the effect of instructional prompts and Multi-Task Learning (MTL) has not been systematically studied in the biomedical domain. Motivated by this, this paper explores the impact of instructional prompts for biomedical MTL. We introduce the BoX, a collection of 32 instruction tasks for Biomedical NLP across (X) various categories. Using this meta-dataset, we propose a unified model termed In-BoXBART, that can jointly learn all tasks of the BoX without any task-specific modules. To the best of our knowledge, this is the first attempt to propose a unified model in the biomedical domain and use instructions to achieve generalization across several biomedical tasks. Experimental results indicate that the proposed model: 1) outperforms the single-task baseline by ~3% and multi-task (without instruction) baseline by ~18% on an average, and 2) shows ~23% improvement compared to the single-task baseline in few-shot learning (i.e., 32 instances per task) on an average. Our analysis indicates that there is significant room for improvement across tasks in the BoX, implying the scope for future research direction.

1 Introduction

The introduction motivates a unified approach to biomedical NLP because task-specific models are costly and biomedical cross-task generalization had not been systematically studied. It introduces the BoX instruction-task collection and In-BoXBART, then reports comparisons with single-task and instruction-free multitask baselines.

1 Introduction

  • Task-specific biomedical models are computationally expensive and time-consuming, while generalization across biomedical NLP tasks had not been systematically studied.The introduction presents a single model for multiple biomedical tasks as a needed alternative.Source: §1 · p. 1
  • In-BoXBART outperforms the single-task baseline by ~3% and the multitask baseline without instructions by ~18%; with 32 instances per task, it outperforms the single-task baseline by 23.33%.The paper links the few-shot setting to the cost and time required to obtain biomedical annotations.Source: §1 · p. 2
  • The paper introduces BoX, a collection of 32 biomedical instruction tasks, and proposes In-BoXBART, an instruction-tuned BART model trained on the meta-dataset.Tasks are processed in a unified format for sequence-to-sequence models.Source: §1 · p. 2

2 Related Work

Related work describes multi-task learning through shared task formats and instruction learning through task decomposition, prompt tuning, and instruction refinement. These lines of work include biomedical text-to-text modeling and studies of generalization to unseen tasks.

2 Related Work

3 BoX

BoX brings biomedical NLP datasets together as a nine-category benchmark for multi-task learning, reframing them as text-generation tasks with human-authored instructions. Its unified instruction schema and task-specific construction process cover 32 instruction tasks.

3 BoX

  • BoX combines 29 biomedical NLP datasets into a multi-task benchmark spanning nine categories and 32 instruction tasks.The datasets are reframed as text-generation tasks.Source: §3 · p. 3

3.1 Tasks

  • The benchmark also includes question answering, binary relation extraction, systematic review, sentiment analysis, and document classification tasks.QA includes yes/no, factoid, and list questions; the relation-extraction tasks omit relation-type labels.Source: §3.1 · p. 4
  • Tasks cover varied clinical and literature applications, including risk-factor identification, named-entity recognition, de-identification, and POS tagging.Examples include identifying risk factors over time in diabetic patients and extracting biomedical entities from literature.Source: §3.1 · pp. 3–4

3.2 Biomedical Instructions

  • Biomedical Instructions describe tasks in natural language and accompany task instances; each dataset in BoX receives a manually completed BI verified by other authors.Instructions were checked for grammar, typos, ambiguity, and repetition; positive examples are selected from training data and removed from it.Source: §3.2 · p. 4, §3.2.2 · p. 5

3.2.1 Unified Schema

  • Each BI follows a unified schema comprising a definition, prompt, positive examples, and input/output instances.Definitions explain the task, prompts briefly state it, and examples include input/output pairs with explanations; generally, two to three examples are provided.Source: §3.2.1 · pp. 4–5

3.2.2 Construction of BI

  • To construct the task data, dataset instances are converted to text-to-text input/output pairs and appended to the instruction schema.Instructions are created for each dataset, with each BI completed by one author and verified by others.Source: §3.2.2 · p. 5

4 Problem Setup and Models

The setup contrasts task-specific learning with a single model trained across tasks using biomedical instructions. The proposed In-BoXBART combines each task's instructions and training data, while the baselines omit instructions in the multi-task setting or train separately by task.

4.1 Problem setup

  • Multi-task learning combines training data and biomedical instructions across tasks so one model learns to map instructions and inputs to outputs.The model is evaluated on task-specific instances.Source: §4.1 · p. 5

4.2.1 Baselines

  • The single-task baseline trains a BART model on each BoX task and evaluates it on that same task.Source: §4.2.1 · p. 5
  • Vanilla-BoXBART trains one model on the combined task data without appending biomedical instructions and evaluates it on each task.Source: §4.2.1 · p. 5

4.2.2 Proposed Model

  • The proposed In-BoXBART fine-tunes BART on combined task data after converting each instruction and input into text and appending the instruction before the input.Instructions include definitions and positive examples; the model maps the encoded instruction and input to an output.Source: §4.2.2 · pp. 5–6, §4.2 · p. 5

5 Experiments and Analysis

Experiments compare instruction-tuned In-BoXBART with single-task and instruction-free multitask baselines, test sampling and few-shot settings, and analyze where instructions help or remain challenging.

5.1 Experimental Setup

  • Few-shot subsets begin with 32 randomly selected, balanced instances per task and scale to 100, 1k, or 4k, retaining all data for smaller tasks.The BoX averages 6k instances per task; samples exceeding BART's 1,024-token input limit are excluded from task creation and validation and testing.Source: §5.1 · p. 6

5.2 Results and Findings

  • Over-sampling achieves 73.49 average Rouge-L, compared with 69.62 for under-sampling and 70.23 for average-sampling.The authors attribute the weaker under- and average-sampling results to loss of training samples and use over-sampling for the main results.Source: §5.2 · p. 6
  • Vanilla-BoXBART uses 110 million parameters rather than 32x110 million, but its average Rouge-L is 14.96% below single-task models.In-BoXBART has the same complexity as Vanilla-BoXBART.Source: §5.2 · pp. 7–8
  • At 32 instances per task, In-BoXBART achieves 60.64% Rouge-L versus 37.31% for single-task models, a 23.33% improvement.Figure 5 compares average Rouge-L across instruction tasks by average training instances per task.Source: §5.1 · p. 6, Figure 5 · p. 8, §5.2 · p. 8
  • On average, In-BoXBART exceeds Vanilla-BoXBART by 17.94% and single-task models by 2.98%.The comparison reports average performance across tasks; Table 2 compares Rouge-L and F1-Score.Source: §5.2 · pp. 6, 8, Table 2 · p. 7

5.3 Analysis

  • In-BoXBART outperforms baselines in NER, de-identification, relation extraction, semantic role labeling, and risk-factor identification; it is no better than single-task models in four other categories.The latter categories are QA, POS tagging, sentiment analysis, and document classification.Source: §5.3 · p. 8
  • Multi-class tasks and answer generation from context are generally harder; CRAFT and BioNLP13CG have six entity types and lower NER performance than other tasks.De-identification performance is 50.82% despite instructions improving it.Source: §5.3 · p. 8
  • In few-shot settings with 32 or 100 instances per task, NER, de-identification, QA, sentiment analysis, and risk-factor identification show larger average gains over baselines.Source: §5.3 · p. 8

6 Discussion

The discussion identifies instruction design, long-context inputs, multiclass tasks, and gaps from state-of-the-art performance as open challenges for biomedical MTL.

6 Discussion

  • Biomedical instructions could benefit from domain-specific information and domain-expert knowledge, making instruction design a research direction.The discussion asks whether better instructions can be designed.Source: §6 · p. 8
  • Long EHR and discharge-summary inputs can exceed BERT and BART limits, making it difficult to encode additional information in prompts or instructions.Longformer and T5-like models are proposed as possible solutions.Source: §6 · pp. 8–9
  • Multiclass instructions cannot include one example per class when the resulting input exceeds model limits; using fewer labels or omitting examples can introduce label bias or reduce performance.A methodology for handling multiclass classification is proposed as future work.Source: §6 · p. 9
  • For some of the 21 BoX instruction tasks compared, In-BoXBART remains far from state-of-the-art single-task systems, indicating room for further research.The comparison is described as preliminary.Source: §6 · p. 9

7 Summary and Conclusions

The paper presents BoX and In-BoXBART for biomedical instruction-based MTL, reporting gains over two baselines and in few-shot learning. It concludes that the benchmark and results can encourage research into unified biomedical NLP models.

7 Summary and Conclusions

  • In-BoXBART outperforms the single-task baseline by ∼3% and Vanilla-BoxBART by ∼18%.The paper presents this as evidence of the impact of instructions in biomedical MTL.Source: §7 · p. 9
  • BoX is introduced as a benchmark dataset comprising 32 instruction tasks across biomedical NLP domains.The paper describes it as the first benchmark dataset of this kind in the biomedical domain.Source: §7 · p. 9
  • The authors report effective few-shot performance and suggest the work may encourage future research into unified models for biomedical NLP.They connect the few-shot setting to the difficulty of obtaining large annotated biomedical datasets.Source: §7 · p. 9

A Statistics of Instruction Tasks

The appendix describes the data handling and unified text-to-text preparation used for BoX experiments.

A Statistics of Instruction Tasks

B Instruction Tasks and Examples

C Systematic Review Datasets

The study uses systematic-review screening data developed from clinical reviews and labeled by physicians.

C Systematic Review Datasets

  • The study uses data from five manually conducted clinical systematic reviews: Hormone Replacement Therapy, Cooking, Accelerometer, Acromegaly, and COVID.The dataset was developed from six reviews, but this study uses five.Source: C Systematic Review Datasets · p. 12
  • An experienced librarian conducted the initial bibliographic searches with guidance from the principal investigators.Searches included PubMed, PubMed Central, Embase, EBM Reviews, and Ovid MEDLINE.Source: C Systematic Review Datasets · p. 12
  • Two physicians with domain expertise classified each search-result article as Include or Exclude using its title and abstract.Source: C Systematic Review Datasets · p. 12
  • When the two annotators disagreed, the positive Include label was preferred.Source: C Systematic Review Datasets · p. 12

D Few-Shot Learning results

This section presents few-shot learning results for all BoX instruction tasks in Table 5.

D Few-Shot Learning results

  • Table 5 reports few-shot learning results for all BoX instruction tasks.The supplied passage does not provide the table’s numerical results.Source: D Few-Shot Learning results · p. 13

E State-of-the-art results

The section compares Vanilla-BoXBART and In-BoXBART with state-of-the-art results across 21 tasks. It describes how reference results were selected, but the supplied passages do not report comparative outcomes.

E State-of-the-art results

  • The state-of-the-art comparison covers 21 tasks, using the best performance gathered for each task.For ChemProt and DDI, the comparison uses base language models rather than large models for a fair comparison.Source: E State-of-the-art results · p. 13
  • Reference results for all 15 NER datasets come from Banerjee et al. (2021), while the HoC reference is the best system submitted to Stubbs et al. (2015).The passage also names Nentidis et al. (2020) for BioASQ-8b and Peng et al. (2019) for ChemProt, DDI, and HoC.Source: E State-of-the-art results · p. 13
  • Table 6 compares Vanilla-BoXBART and In-BoXBART with state-of-the-art results for each task, with all results reported in percent.The supplied passages identify the comparison but do not provide table values or winners.Source: E State-of-the-art results · p. 13, Table 6 · p. 17

Related papers

Loading 2204.07600v1…