PaperlayerView exact revision on arXiv

Source-linked AI summary

Large language models as synthetic clinical experts to inform longitudinal rare-disease modeling

Clemens Schächter, Astrid Pechmann, Janbernd Kirschner, Jan Hasenauer, Harald Binder

arXiv:2608.16507v1cs.AI

TL;DR

Longitudinal rare-disease modeling has limited data, while eliciting and formalizing clinical expertise for model fitting is difficult. This paper uses LLM-derived clinical judgments, distilled into a differentiable surrogate, to steer a variational autoencoder toward clinically consistent reconstructions. In SMA motor-function data, this supervision reduced disagreement between observed and reconstructed judgment distributions while HINE-2 reconstruction errors remained in a similar range at moderate supervision weights.

  • Problem

    In rare-disease modeling, incorporating clinically important distinctions into model fitting is difficult because expert input takes time and is hard to formalize.

    Source: §1 · p. 2, §4 · p. 16

  • Method

    The approach uses LLM-assigned clinical labels to train a differentiable surrogate that steers a generative representation model toward reconstructions with matching clinical interpretations.

    Source: §2.2 · p. 8, §4 · p. 16

  • Results

    Synthetic-expert supervision reduced disagreement between observed and reconstructed judgment distributions, while HINE-2 reconstruction errors remained in a similar range at moderate supervision weights.

    Source: §4 · p. 16

  • Takeaways & Limitations

    The findings support synthetic-expert supervision as a way to incorporate clinical judgment into disease-progression modeling, not as a replacement for clinical expertise.

    Source: §4 · p. 17

    The approach depends on the prompts, label definitions, textual renderings, and LLM used; systematic errors in its judgments can shape the learned representation, so outputs require auditing.

    Source: §4 · p. 16

Abstract

from arXiv · show

Due to the limited amount of information, modeling longitudinal rare-disease data can benefit from integrating clinical knowledge. Yet, elicitation of expert knowledge and formalization for model fitting is challenging, in particular due to limited time of clinical experts. To nevertheless make domain knowledge accessible during model fitting, we use large language models (LLMs) as synthetic clinical experts to supervise a variational-autoencoder-based approach that learns low-dimensional latent summaries of visit-level observations. Specifically, LLMs are queried offline on textual descriptions of patient observations to obtain judgments, e.g., the suspected clinical category. To improve the variational autoencoder fit, we train a differentiable surrogate model on these judgments and augment the loss function to encourage reconstructions that preserve the clinical-label distribution of their corresponding input profile. In an application to longitudinal motor-function assessments from children with spinal muscular atrophy, we map visit-level clinical profiles to low-dimensional representations that are linked by a multivariate mixed-effects model. The synthetic expert loss discourages reconstructions that remain numerically close in data space but alter the clinical interpretation of the reconstructed motor function profile, such as by crossing a disease-type boundary. We thus reduced disagreement between original and reconstructed SMA type labels from about 11 to 7 percent. Furthermore, informing the latent representation by the synthetic expert improved prediction of motor function milestones compared with unsupervised latent representations and a data-level baseline. These results suggest that incorporating LLMs into model fitting can make clinical knowledge available to representation learning and improve clinical faithfulness for longitudinal rare-disease data.

1 Introduction

This study uses LLM-derived clinical judgments as synthetic-expert supervision directly in a generative model’s fitting objective. In longitudinal SMA motor-function data, it aims to preserve clinical interpretations in latent representations and assess their usefulness for milestone prediction.

Introduction

  • Clinical knowledge is difficult and time-consuming to formalize, while likelihood, prediction, and reconstruction objectives may fail to preserve clinically relevant structure.Rare-disease interpretation relies on clinical experience and context, and sustained clinician–statistician exchange can introduce miscommunication (Daee et al. 2017; Mikkola et al. 2023; Oakden-Rayner et al. 2019).Source: §1 · pp. 2–3
  • SMA type assessment depends on age and expected milestones, so superficially similar motor-function profiles may receive different clinical labels that are difficult to formalize.The assessment records ordinal items spanning abilities such as head control, sitting, crawling, standing, and walking.Source: §1 · pp. 3–4
  • A cVAE learns visit-level latent variables linked across observation times by a multivariate mixed-effects model, reducing parameters for many test items in small rare-disease cohorts.Reconstructions can be numerically close yet cross a clinically meaningful boundary, while larger numerical errors may retain the same functional interpretation; the approach builds on prior VAE-based representation methods (Ong et al. 2024; Schächter et al. 2025).Source: §1 · p. 4
  • Pairwise LLM questions assign original and reconstructed profiles to candidate SMA types; consistency checks and a learned neural-network surrogate provide differentiable gradients for VAE training.Pairwise judgments cover all combinations of SMA type labels, and inconsistencies are identified before they guide the cVAE.Source: §1 · p. 4
  • Unlike weak-supervision and distillation approaches that use LLM labels for downstream models, this method uses LLM judgments to steer generative-model fitting through a surrogate.LLM judgments can be costly, stochastic, and non-differentiable, motivating conversion into an auxiliary model before optimization (Hsu and Roberts 2025; Smith et al. 2022; Hsieh et al. 2023; Hinton, Vinyals, and Dean 2015; Christiano et al. 2017; Ouyang et al. 2022).Source: §1 · pp. 4–5
  • LLM-derived judgments directly supervise generative representation-model fitting to discourage reconstructions that alter clinical interpretation.The study extends LLM-based supervision beyond downstream label generation and investigates clinical faithfulness and milestone-prediction utility in longitudinal SMA data.Source: §1 · pp. 3, 5

2 Methods

The method uses offline LLM pairwise judgments to derive clinical labels, then trains a differentiable surrogate to preserve those judgments in reconstructed profiles. A conditional VAE maps longitudinal visits to low-dimensional representations linked through a multivariate mixed-effects model.

2.1 Synthetic-expert label generation

  • For each rendered clinical profile, the LLM chooses between ordered pairs of candidate labels, and ordinally consistent pairwise answers are aggregated into a synthetic-expert label.Input text can include relevant visit context and descriptions of clinical measurements; ordinal consistency is checked against the ordered clinical categories.Source: §2.1 · pp. 6–7

2.2 Synthetic-expert supervision

  • A surrogate neural network approximates the LLM-derived label distribution, and a Jensen-Shannon divergence penalty encourages reconstructed profiles to retain the observed profile’s clinical interpretation.The surrogate is held fixed during model optimization, while gradients pass through it with respect to the reconstructed-profile input.Source: §2.2 · p. 8

2.3 Linking visit-level latent representations with a multivariate mixed-effects model

  • The conditional VAE compresses high-dimensional visits into low-dimensional representations linked by a multivariate mixed-effects model, with encoder-decoder and latent-model parameters updated alternately.The encoder and decoder are conditioned on visit context; the learned representations can be used for downstream time-to-event modeling.Source: §2.3 · pp. 9–10

3 Results

Synthetic-expert supervision made cVAE reconstructions more consistent with LLM-derived SMA categories while largely preserving numerical reconstruction quality, and supervised latent representations generally improved milestone prediction over unsupervised latent representations. LLM judgments also varied with age and motor-function profile, supporting their use as a developmental-context-sensitive signal.

3.1 Application to spinal muscular atrophy data

  • The cVAE modeled age-conditioned HINE-2 visit profiles and linked their low-dimensional latent representations across observation times in SMArtCARE children with SMA.Synthetic-expert supervision targeted four categories: SMA types 1–3 and presymptomatic motor development.Source: §3.1 · pp. 11–12

3.2 Obtaining a differentiable synthetic-expert surrogate on SMA data

  • Gemma-4-31B yielded consistent final labels for 89.9% of samples, with label-order agreement for 97.8% of pairwise comparisons and 92.7% of final SMA type labels.For this model, self-reported uncertainty was 9.6 times higher in order-unstable than stable comparisons.Source: §3.2 · p. 13
  • Surrogate accuracy exceeded 90% in held-out folds for the largest model in each family under fivefold cross-validation.Contradictory pairwise-answer profiles were excluded from training, and the surrogate used two heads to reproduce both label-order choices.Source: §3.2 · p. 13
  • The surrogate captured age-dependent clinical interpretation: identical motor-function profiles received different class probabilities as age varied, and profiles with sitting as their highest function shifted from presymptomatic labels at young ages to SMA type 2.Standing without walking was classified as SMA type 2 or 3, while walking profiles were predominantly presymptomatic.Source: §3.2 · pp. 13–14

3.3 Consequences for consistency and reconstruction loss

  • Increasing synthetic-expert supervision reduced disagreement between observed and reconstructed profiles in surrogate judgment distributions and expected SMA type labels, while average HINE-2 reconstruction error stayed close to baseline.Profiles whose unsupervised reconstructions crossed an SMA type boundary showed a noticeable increase in reconstruction quality, alongside small deterioration for profiles without label disagreement.Source: §3.3 · p. 14

3.4 Milestone prediction with Cox regression

  • Supervised latent representations generally improved milestone-prediction performance over the unsupervised cVAE latent baseline and data-level baselines, with moderate weighting appearing to balance supervision and the original cVAE loss.For larger weights such as λSE = 100, performance decreased on some metrics; λSE = 25 appeared to provide a balance.Source: §3.4 · p. 15

4 Discussion

The discussion presents LLM-derived clinical judgments, distilled into a differentiable surrogate, as a way to incorporate clinical knowledge into longitudinal rare-disease modeling. It reports improved clinical consistency and supportive milestone prediction while emphasizing judgment quality, validation, and uncertainty as important limitations.

  • The framework uses LLM-assigned labels on textual observations to train a differentiable surrogate that steers a generative model away from numerically close but clinically misinterpreted reconstructions.It addresses the difficulty of translating clinical distinctions into explicit formulas, priors, or constraints.Source: §4 · pp. 15–16
  • Judgment consistency depended strongly on LLM size and family, so the authors recommend prioritizing LLM quality before offline label generation.The judgments were extracted from spinal muscular atrophy motor-function data using pairwise-comparison prompts.Source: §4 · p. 16
  • Synthetic-expert supervision improved agreement between observed and reconstructed clinical judgments while maintaining similar HINE-2 reconstruction errors at moderate weights, and milestone-prediction results were supportive.Because SMA type categories are closely tied to sitting, standing, and walking milestones, improved consistency in type interpretation also encouraged the latent representation to encode milestone-relevant information.Source: §4 · p. 16
  • Synthetic-expert errors can become part of the objective, and consistency checks cannot establish clinical correctness, so outputs require careful audit before clinical deployment.Uncertainty is currently mainly diagnostic; future extensions could down-weight uncertain labels or use softer target distributions.Source: §4 · pp. 16–17
  • The surrogate makes gradient-based training feasible but can blur boundaries in sparse or ambiguous HINE-2 regions; overly high supervision weights can harm reconstruction and downstream prediction.The supervision weight should be selected as a hyperparameter using validation criteria that reflect both numerical reconstruction and intended clinical use.Source: §4 · p. 17
  • The authors frame synthetic-expert supervision as a practical interface between clinical reasoning and statistical modeling, not a replacement for clinical expertise.Future work should strengthen clinical validation and propagate synthetic-expert uncertainty into the loss.Source: §4 · p. 17

Ethics statement

The study used anonymized SMArtCARE registry data reviewed by the University of Freiburg Ethics Committee, with registry authorization and local LLM inference in secure computing environments.

Ethics statement

  • The University of Freiburg Ethics Committee reviewed the use of anonymized SMArtCARE data, the registry authorized its provision, and LLM inference ran locally in secure computing environments.Ethics reference no. 22-1430-S1-retro.Source: Ethics statement · p. 18

Conflicts of interest

The authors report one editorial role and no conflicts of interest for the remaining authors.

  • Harald Binder serves as Guest Editor of the Biostatistics special collection Statistical Foundations of AI and Real-World Evidence Generation; the remaining authors declare no conflicts of interest.Source: Conflicts of interest · p. 18

Supplementary materials for: Large language

The supplementary materials accompany “Large language models as synthetic clinical experts to inform longitudinal rare-disease modeling.”

A Data preprocessing and hyperparameters · A.1 Dataset

The study used longitudinal HINE-2 assessments from children with SMA in the SMArtCARE registry, preprocessing ordinal item responses for latent mixed-effects modeling.

A.1 Dataset

  • Patients under 12 years were included, and each was required to have more than four complete HINE-2 visits.Visits with missing values were excluded.Source: §A.1 · p. 27
  • HINE-2 profiles contained eleven categorical motor-function items, whose responses were mapped to ordered numerical levels and thermometer-encoded for neural-network inputs.Source: §A.1 · p. 27
  • The registry tracks motor function development, motor milestone achievement, and treatment history in children diagnosed with spinal muscular atrophy.Source: §A.1 · p. 27
  • The eleven HINE-2 items assessed limb function, reaching, head control, sitting, grasp, kicking, rolling, crawling, standing, and walking.Source: §A.1 · p. 27
  • The dataset contained exactly 13,000 observations from 994 patients, with a median of 12 visits per patient.Source: §A.1 · p. 28
  • Median age at first observation was 1.2 years, and median follow-up duration was 3.5 years.Source: §A.1 · p. 28

A.2 Synthetic-expert label generation and surrogate training

The authors generated synthetic HINE-2 profiles, elicited labels through pairwise LLM comparisons, and trained a surrogate classifier to reproduce retained labels from age and profile scores.

A.2 Synthetic-expert label generation and surrogate training

  • A two-dimensional age-conditional VAE generated 20,000 unique profiles, with ages jittered by up to 0.25 years.The VAE was fitted to real HINE-2 data, and profiles were sampled from its decoder.Source: §A.2 · p. 28
  • Profiles were rendered as age-tagged motor-function descriptions using HINE-2 item wording [1] and score-to-text mappings shown in Table S1.For each item, the sentence corresponding to its observed score was selected and concatenated with the other item descriptions and patient age.Source: §A.2 · p. 28, Table S1 · pp. 28–29
  • Each profile was compared pairwise across SMA types 1–3 and presymptomatic motor development, with label order reversed to create two comparison panels per profile.The prompt template for each pairwise comparison is shown in Table S2.Source: §A.2 · p. 29, Table S2 · p. 29
  • Pairwise answers were aggregated using the labels’ ordinal order; contradictory or invalid response patterns were excluded from surrogate training.The ordinal order was SMA type 1 < SMA type 2 < SMA type 3 < presymptomatic motor development.Source: §A.2 · pp. 29–30
  • The surrogate classifier used age and thermometer-encoded HINE-2 profiles to reproduce retained labels, with two heads for the two label orders and averaged results.Each head reproduced answers for one label order.Source: §A.2 · p. 30

A.3 Hyperparameters

The study used a two-dimensional conditional VAE and a latent mixed-effects model, with neural-network, optimization, and supervision settings specified for fitting.

A.3 Hyperparameters

  • Neural networks used one 100-unit ReLU hidden layer, and the surrogate was trained on 41 input units with cross-entropy, Adam defaults, and 50 epochs.The 41 inputs comprised thermometer-encoded HINE-2 profiles and age.Source: §A.3 · p. 30
  • The cVAE used a two-dimensional age-conditioned latent space, an ordinal categorical decoder, Adam at learning rate 0.01 for 20 alternating epochs, β = 0.5, η = 5, and λSE ∈{0, 10, 25, 50, 100}.Source: §A.3 · p. 30
  • The latent mixed-effects model used patient-specific random intercepts and slopes, with fixed effects including clinical and treatment variables and age interactions, following previous work [2].Treatment-time variables were zero before treatment or switching; parameters were re-estimated after each cVAE epoch with L-BFGS at learning rate 0.1.Source: §A.3 · p. 30

A.4 Milestones and Cox regression

The section defines milestone events and censoring for Cox regression, then describes time-varying model inputs and patient-grouped cross-validation. Predictions are assessed at a one-year horizon using discrimination, error, and calibration metrics.

A.4 Milestones and Cox regression

  • Exact milestone-attainment ages were uncensored events; unknown exact ages were right-censored, while milestones achieved before observation without an exact age were left-censored and excluded.Independent sitting, standing, and walking were the registry endpoints; Table S3 gives their definitions and observed-event scope.Source: §A.4 · pp. 30–31, Table S3 · p. 31
  • Visits before each milestone event or censoring were converted to age-scale Cox intervals, and linear Cox models of latent representations were compared across synthetic-expert supervision weights using patient-grouped five-fold cross-validation.Evaluated feature sets were the HINE-2 total score, that score plus fixed clinical covariates, and latent encoder coordinates.Source: §A.4 · p. 31
  • One-year landmark predictions were evaluated with censoring-weighted time-dependent AUC, weighted Brier score, and expected calibration error across 10 quantile-based bins.Visits censored before one year received inverse-probability-of-censoring weights, and rows were weighted inversely by each patient's number of landmark visits per milestone.Source: §A.4 · pp. 31–32

Related papers

Loading 2608.16507v1…