PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal LLMs for health grounded in individual-specific data

Anastasiya Belyaeva, Justin Cosentino, Farhad Hormozdiari, Krish Eswaran, Shravya Shetty, Greg Corrado, Andrew Carroll, Cory Y. McLean, Nicholas A. Furlotte

arXiv:2307.09018v2q-bio.QMcs.LG

TL;DR

Health data are high-dimensional and multimodal, and relevant inputs may not map clearly into text. HeLM maps non-text modalities into an LLM’s token embedding space for disease-risk prediction, with UK Biobank results showing competitive performance against classic ML and occasional outperformance.

  • Problem

    Health data are high-dimensional and multimodal, and relevant inputs may not map clearly into text, while potential applications range from clinician assistants to wellness chatbots.

    Source: §1 · p. 2, §4 · p. 14

  • Method

    HeLM maps non-text health modalities into an LLM’s token embedding space and uses the LLM to estimate disease risk.

    Source: §1 · pp. 3–4

  • Results

    On UK Biobank disease-risk tasks, HeLM was competitive with classic machine-learning methods and sometimes outperformed them, matching logistic regression and XGBoost in reported comparisons.

    Source: §1 · p. 4, §4 · p. 14

  • Takeaways & Limitations

    The findings support the promise of using complex multimodal health data to quantify underlying health risks with LLMs.

    Source: §4 · p. 14

    Restricting UK Biobank analyses to European ancestry reduces phenotypic heterogeneity but creates bias in the dataset and subsequent analyses.

    Source: §2.3 · p. 6

Abstract

from arXiv · show

Foundation large language models (LLMs) have shown an impressive ability to solve tasks across a wide range of fields including health. To effectively solve personalized health tasks, LLMs need the ability to ingest a diversity of data modalities that are relevant to an individual's health status. In this paper, we take a step towards creating multimodal LLMs for health that are grounded in individual-specific data by developing a framework (HeLM: Health Large Language Model for Multimodal Understanding) that enables LLMs to use high-dimensional clinical modalities to estimate underlying disease risk. HeLM encodes complex data modalities by learning an encoder that maps them into the LLM's token embedding space and for simple modalities like tabular data by serializing the data into text. Using data from the UK Biobank, we show that HeLM can effectively use demographic and clinical features in addition to high-dimensional time-series data to estimate disease risk. For example, HeLM achieves an AUROC of 0.75 for asthma prediction when combining tabular and spirogram data modalities compared with 0.49 when only using tabular data. Overall, we find that HeLM outperforms or performs at parity with classical machine learning approaches across a selection of eight binary traits. Furthermore, we investigate the downstream uses of this model such as its generalizability to out-of-distribution traits and its ability to power conversations around individual health and wellness.

1 Introduction

Personalized health tasks require LLMs to use complex, individual-specific multimodal data, a capability that remains understudied. HeLM maps non-text health data into an LLM’s token space for disease-risk estimation, while the paper also evaluates serialized inputs and prompting approaches.

1 Introduction

  • Health applications need models to use complex, individual-specific multimodal data to understand underlying health risks and condition responses on an individual’s risk profile.The paper notes that this use of multimodal individual-specific information remains understudied despite work on generalist medical LLMs.Source: §1 · p. 2
  • Text prompts can represent simple risk factors, but incorporating images, time series, or rich structured data—and combining modalities—remains unclear.The paper frames these challenges as a reason to investigate multimodal approaches for health.Source: §1 · pp. 2–3
  • Foundation LLMs may offer health-related prior knowledge, flexible handling of missing data, and conversational interaction, while traditional health multimodal methods lack some potential advantages of LLM-based methods.The paper also notes that multimodal LLM methods had not yet been applied to health in the related work it discusses.Source: §1 · p. 3
  • Soft-prompt tuning brings serialized health features to comparable or often equivalent performance with logistic regression and XGBoost, while zero-shot and few-shot performance is generally better than random.For some diseases, including diabetes, zero-shot and few-shot approaches perform surprisingly well against those baselines; performance degrades as the number of serialized input features increases.Source: §1 · p. 4
  • HeLM maps non-text modalities into the LLM’s token embedding space and matches or sometimes outperforms classical machine-learning methods for disease-risk estimation.The evaluated non-text modalities include tabular data and spirogram curves; the model is also explored for downstream conversational tasks.Source: §1 · p. 4

2 Methods

The methods compare serialized tabular inputs with a multimodal approach that maps non-text data into the LLM’s token embedding space. Experiments use UK Biobank clinical features, spirometry, and disease labels, with the dataset restricted to a European-ancestry subpopulation.

2.1 LLMs with Tabular Data

  • For tabular prediction, column names and values are serialized in a JSON-like prompt, and the model’s log-likelihood for “yes” versus “no” completion serves as a risk score.The serialized-input approach is evaluated in zero-shot, few-shot, and soft-prompt tuning settings; soft prompts are trained with labeled examples while the language model remains frozen.Source: §2.1 · pp. 4–5

2.2 Multimodal LLMs for health: HeLM

  • HeLM maps each non-text modality to a sequence of vectors in the language-token embedding space using a separate encoder, then processes the combined inputs with a pretrained LLM.The experiments use six vectors per modality and keep the pretrained LLM weights frozen while training modality-specific encoders.Source: §2.2 · p. 5

2.3 UK Biobank Dataset Preparation

  • The UK Biobank dataset provides clinical features, spirometry, and disease labels; phenotype labels combine medical records and self-reports, and inputs include demographic, clinical, laboratory, and wellness features.Spirometry curves are processed into normalized flow-volume spirograms, and the analysis uses a European-ancestry subpopulation, a choice the authors say creates bias in the dataset and subsequent analyses.Source: §2.3 · pp. 5–6

3 Experimental Results

The experiments compare LLM prompting and HeLM with classical methods for disease-risk prediction, then test quantitative-data encoding, multimodal asthma prediction, out-of-distribution traits, and conversational generation. Across these settings, HeLM matches or exceeds baselines in reported comparisons, while the conversation experiments remain exploratory.

3.1 Quantifying disease risk using zero-shot, few-shot, and soft-prompt tuning

  • At least one prompting approach was competitive with classical baselines on most tasks, and zero-shot and few-shot models sometimes performed well for hypertension and diabetes.These prompting approaches used serialized health data; the authors suggest their performance may reflect prior knowledge about age, sex, BMI, and disease likelihood.Source: §3.1 · p. 7, Table 1 · p. 8
  • For diabetes, hypertension, and stroke, LLM feature effects generally aligned with logistic regression and XGBoost; migraine required soft-prompt tuning for alignment with logistic regression.The authors hypothesize that soft-prompting learns a linear mapping of input features.Source: §3.1 · pp. 7–8
  • For expanded feature sets, text-serialized zero-shot prompting could perform worse than using fewer features, and soft-prompt tuning did not restore baseline performance.The authors interpret this as evidence that text serialization does not fully capture complex input signal, motivating direct embedding of quantitative data.Source: §3.1 · pp. 8–9

3.2 Encoding quantitative data using HeLM

  • Directly encoding quantitative tabular features let HeLM match the best baseline methods and outperform them for all-cause mortality, hypertension, and myocardial infarction.HeLM scores correlated strongly with logistic regression scores (Spearman = 0.70–0.87).Source: §3.2 · p. 9, Table 2 · p. 10

3.3 Estimating asthma risk using multiple modalities

  • AUROC increased from 0.49 ± 0.02 for a tabular-only soft-prompt LLM to 0.75 ± 0.01 for HeLM using tabular data and a spirogram.AUPRC increased from 0.14 ± 0.01 to 0.38 ± 0.02; HeLM performed on par with a linear combination of ResNet18 1D spirogram and tabular data.Source: §3.3 · pp. 11–12, Figure 2 · p. 11

3.4 Using HeLM for out-of-distribution traits

  • For four of five tested out-of-distribution traits, HeLM—untrained on those traits—performed on par with trait-specific logistic regression.The authors suggest HeLM may leverage risk-factor information learned from related traits, but say further research is needed to understand its OOD scores.Source: §3.4 · p. 12, Table 3 · p. 13

3.5 Natural language generation

  • PaLM-2 recommendations varied across 110 examples spanning asthma-risk levels when HeLM risk estimates were supplied as text.Directly conditioning HeLM on spirogram embeddings produced reasonable sample recommendations, but answers also included yes/no labels, likely reflecting the prompt format.Source: §3.5 · pp. 12–13

4 Discussion

HeLM maps non-text health data into an LLM’s token embedding space and, on UK Biobank data, is competitive with classical machine-learning approaches, sometimes outperforming them. The discussion identifies limits in the modalities and conversational capabilities evaluated, as well as unresolved concerns around explanations, uncertainty, and demographic bias.

4 Discussion

  • On UK Biobank data, HeLM is competitive with classical machine-learning approaches and sometimes outperforms them by mapping non-text modalities into an LLM’s token embedding space.The authors describe this as a step toward using complex multimodal health data to quantify underlying health risks.Source: §4 · p. 14
  • The experiments cover scalar lab values and high-dimensional lung-function data, leaving joint imaging inputs, missing-data behavior, and non-text generation for future study.The authors specifically identify fundus images and cardiac MRI as possible additional modalities.Source: §4 · p. 14
  • Conversational ability degraded after model tuning in the asthma-risk recommendation experiment.The authors say this is consistent with prior observations [35], while noting some evidence that larger models are more robust [11].Source: §4 · p. 14
  • Conversational agents should explain high- or low-risk classifications and quantify uncertainty; sensitive health topics also raise alignment concerns.The discussion notes that some topics lack expert consensus or depend on users’ cultures and preferences.Source: §4 · p. 14
  • The study did not address possible differential performance across demographic groups from computational, systemic, or human biases.The authors say evaluation and mitigation are important before deploying multimodal health LLMs.Source: §4 · p. 14

A Data Availability

The paper states that phenotypes and genotypes are available to approved projects through the UK Biobank study.

A Data Availability

  • Approved projects can access phenotypes and genotypes through the UK Biobank study.The research was conducted under Application Number 65275.Source: A Data Availability · p. 19

B Definitions of UK Biobank phenotypes

The appendix describes how the study selected its European-ancestry UK Biobank subset and defined phenotype case-control status. Phenotype entries are associated with UK Biobank data-field IDs and coding sources.

B Definitions of UK Biobank phenotypes

  • The European-ancestry subset was selected using self-reported British ancestry and distance from the British-set medoid in 15-dimensional genetic principal-component space.Individuals with a distance below 40 were selected; the cutoff corresponds to the 99th percentile for people self-identifying as British or Irish.Source: B Definitions of UK Biobank phenotypes · p. 19
  • The phenotype table lists UK Biobank Data-Field IDs and coding sources, including ICD-9 hospital inpatient codes from field 41271 and ICD-10 general-practitioner note codes from field 42040.Self-reported status is defined using field 20002.Source: Table S1 · p. 20
  • When a phenotype has multiple label sources, the study combines them with a binary OR to determine case-control status.Hypertension is given as an example with ICD-9, ICD-10, and self-report codes.Source: Table S1 · p. 20

C LLM-based vs. logistic regression and XGBoost feature importances

The appendix compares feature weights across disease-risk scoring methods. It describes regression coefficients as a way to compare the influence assigned to input features.

C LLM-based vs. logistic regression and XGBoost feature importances

  • Fig. S1 compares feature weights for zero-shot, few-shot, soft-prompt, and logistic-regression scores across four phenotypes.Linear regressions predict each score from age, sex, and BMI; their coefficients provide a sense of the weight assigned to each input feature.Source: Figure S1 · p. 21
  • Regression coefficients provide the stated basis for comparing how scoring methods weight age, sex, and BMI.This feature-weight analysis is described for four phenotypes in Fig. S1.Source: Figure S1 · p. 21
  • Fig. S2 compares feature weights between logistic regression, XGBoost, and HeLM.The supplied caption does not report which method assigns greater weight to any feature.Source: Figure S2 · p. 22

D Single task vs. mixture of tasks comparison

Table S2 compares validation performance for HeLM models trained on a mixture of seven diseases versus only the target disease, using an extended feature set.

D Single task vs. mixture of tasks comparison

  • Table S2 compares mixture-trained and target-disease-only HeLM models for all-cause mortality, hypertension, and myocardial infarction.The models use an extended feature set of 14 clinical and wellness features.Source: Table S2 · p. 22
  • Validation performance is reported as mean AUC and AUPRC with 95% confidence intervals across 1,000 bootstrap iterations.Source: Table S2 · p. 22
  • Bold cells mark the best models for each phenotype and input feature set, with significance determined by paired bootstrapping.Source: Table S2 · p. 22

E Differential LLM recommendations based on predicted asthma risk

Figure S3 shows exercise recommendation frequency as a function of predicted asthma risk.

E Differential LLM recommendations based on predicted asthma risk

  • Figure S3 relates exercise recommendation frequency to predicted asthma risk.Source: Figure S3 · p. 23

F Spearman’s rank correlation between different methods

Figures S4 and S5 report Spearman rank correlations between different methods’ scores across seven health outcomes.

F Spearman’s rank correlation between different methods

  • Figures S4 and S5 show Spearman rank correlations between methods’ scores across all-cause mortality, diabetes, hypertension, major depression, migraine, myocardial infarction, and stroke.Source: Figure S4 · p. 24, Figure S5 · p. 25

Related papers

Loading 2307.09018v2…