Large language models as synthetic clinical experts to inform longitudinal rare-disease modeling

Clemens Schächter; Astrid Pechmann; Janbernd Kirschner; Jan Hasenauer; Harald Binder

Source-linked AI summary

Large language models as synthetic clinical experts to inform longitudinal rare-disease modeling

Clemens Schächter, Astrid Pechmann, Janbernd Kirschner, Jan Hasenauer, Harald Binder

arXiv:2608.16507v1Aug 17, 2026cs.AI

TL;DR

Rare-disease modeling has limited data and difficult-to-formalize clinical knowledge. This paper uses LLM-derived judgments distilled into a differentiable surrogate to supervise latent representation learning, improving clinical consistency while preserving similar reconstruction errors.

Problem
Rare-disease modeling is constrained by limited data and the difficulty of eliciting and formalizing expert clinical knowledge for model fitting.
Method
LLMs label textual patient profiles, and a differentiable surrogate uses those judgments to supervise a generative representation model toward clinically consistent reconstructions.
Results
The approach improved consistency between observed and reconstructed clinical judgment distributions while maintaining similar numerical reconstruction errors under moderate supervision weights.
Takeaways & Limitations
Synthetic-expert supervision provides a mechanism for incorporating clinical judgment into longitudinal rare-disease progression modeling without replacing clinical expertise.
Takeaways & Limitations
The method depends on the prompt, label definitions, textual rendering, and LLM quality, so systematic synthetic-expert errors can shape the learned representation.

Abstract

from arXiv · show

Due to the limited amount of information, modeling longitudinal rare-disease data can benefit from integrating clinical knowledge. Yet, elicitation of expert knowledge and formalization for model fitting is challenging, in particular due to limited time of clinical experts. To nevertheless make domain knowledge accessible during model fitting, we use large language models (LLMs) as synthetic clinical experts to supervise a variational-autoencoder-based approach that learns low-dimensional latent summaries of visit-level observations. Specifically, LLMs are queried offline on textual descriptions of patient observations to obtain judgments, e.g., the suspected clinical category. To improve the variational autoencoder fit, we train a differentiable surrogate model on these judgments and augment the loss function to encourage reconstructions that preserve the clinical-label distribution of their corresponding input profile. In an application to longitudinal motor-function assessments from children with spinal muscular atrophy, we map visit-level clinical profiles to low-dimensional representations that are linked by a multivariate mixed-effects model. The synthetic expert loss discourages reconstructions that remain numerically close in data space but alter the clinical interpretation of the reconstructed motor function profile, such as by crossing a disease-type boundary. We thus reduced disagreement between original and reconstructed SMA type labels from about 11 to 7 percent. Furthermore, informing the latent representation by the synthetic expert improved prediction of motor function milestones compared with unsupervised latent representations and a data-level baseline. These results suggest that incorporating LLMs into model fitting can make clinical knowledge available to representation learning and improve clinical faithfulness for longitudinal rare-disease data.

Summary

The paper concerns generative and longitudinal modeling of spinal muscular atrophy using large language models as synthetic clinical experts and variational autoencoders.

The study combines generative models with variational autoencoders for longitudinal modeling.
It applies large language models as synthetic clinical experts to spinal muscular atrophy.

1 Introduction

Rare-disease modeling requires clinical knowledge that is difficult and time-consuming to formalize from scarce, context-dependent data. This work uses LLM-derived clinical judgments directly in a generative representation model to discourage clinically inconsistent reconstructions and assess effects on SMA latent representations.

Clinical experts’ contextual interpretations are difficult to formalize as model structures, priors, constraints, or fitting objectives, especially with scarce rare-disease data.Sustained clinician–statistician exchange can be time-consuming and vulnerable to miscommunication.
LLMs offer a potential solution by providing biomedical knowledge and clinical information through natural-language text-continuation tasks.Prior medical explanation and question-answering applications motivate their use as synthetic clinical experts.
In longitudinal SMA data, motor-function profiles contain multiple ordinal abilities whose clinical interpretation depends on age and expected developmental milestones.Superficially similar profiles may therefore receive different SMA type labels from a clinical expert.
The framework combines a conditional variational autoencoder for visit-level latent variables with a multivariate mixed-effects model linking representations across observation times.The cVAE reduces dimensionality when many motor-function test items must be modeled in small rare-disease samples.
LLM assessments of original and reconstructed profiles are used directly in the fitting objective to discourage clinically inconsistent reconstructions and evaluate latent-representation faithfulness and milestone prediction.Pairwise question-answering assigns profiles to candidate SMA type labels, while the study extends LLM supervision beyond downstream classifier training.

2 Methods

The method uses offline LLM pairwise judgments as synthetic clinical-expert supervision for a conditional variational autoencoder and longitudinal latent mixed-effects model. A differentiable surrogate transfers these judgments into reconstruction training while preserving clinical-label distributions.

Synthetic-expert judgments: LLMs assign forced pairwise clinical-label judgments to rendered patient profiles, with candidate pairs defined over all ordered label combinations.For H clinical categories, the method uses M = H(H − 1) ordered pairs.
Synthetic-expert judgments: Clinical profiles are rendered as text from observed variables and values, optionally augmented with contextual visit information such as age and covariates.Instruction tokens specify the candidate pair, while data tokens represent the clinical observations.
Synthetic-expert judgments: The method checks synthetic-expert responses for label-order stability and, when labels are ordinal, consistency with a single clinical category.In the SMA application, the ordinal sequence is type 1, type 2, type 3, then presymptomatic motor development.
Synthetic-expert supervision: A differentiable surrogate estimates the synthetic-expert label distribution, and Jensen-Shannon divergence between observed and reconstructed profiles is added to the reconstruction loss.The resulting objective is LR(ω; y, ˆy, s) + λSE JS(π, ˆπ), with surrogate parameters fixed during optimization while gradients pass through reconstructed-profile inputs.
Longitudinal representation model: Visit-level observations are encoded into contextual latent variables and linked across time with a multivariate mixed-effects model, whose parameters are updated alternately with the encoder and decoder.Both encoding and decoding are conditioned on visit context because synthetic-expert judgments can depend on information such as age.

3 Results

Results show that larger LLMs produced more internally consistent SMA classifications, while their differentiable surrogates enabled synthetic-expert supervision to preserve clinical interpretations during cVAE reconstruction. Increasing supervision reduced clinical-label discrepancies with only small deterioration in HINE-2 reconstruction quality, and supervised latent representations were evaluated for one-year milestone prediction against established baselines.

Synthetic-expert consistency: Larger LLMs yielded more consistent SMA judgments, with Gemma-4-31B achieving the highest reported internal-consistency and order-stability metrics.Small models such as Qwen-3-0.6B produced patterns close to the random p = 0.5 baseline.
Synthetic-expert consistency: Surrogate accuracy exceeded 90% in held-out folds for the largest model in each family under fivefold cross-validation.The surrogate reproduced final class labels from age and thermometer-encoded HINE-2 profiles while approximating the selected LLM’s differentiable decision mapping.
Reconstruction behavior: Increasing supervision weight reduced both Jensen-Shannon divergence between surrogate judgment distributions and disagreement in expected SMA type labels across LLM families.The supervision signal shifted reconstructions toward retaining the same clinical interpretation as the observed profiles.
Reconstruction behavior: Average HINE-2 reconstruction error stayed close to the unsupervised baseline across supervision weights, with only small deterioration at larger weights.The analysis assessed whether improved SMA-type consistency came at the cost of numerical reconstruction quality.
Prognostic evaluation: Supervised latent representations were evaluated for predicting milestone attainment within one year after held-out landmark visits using time-varying Cox models.Comparisons included the HINE-2 sum score, the sum score plus fixed clinical covariates, and the unsupervised cVAE latent representation, with IPCW Brier score and time-dependent AUC among the metrics.

4 Discussion

The discussion presents synthetic-expert supervision as a practical mechanism for incorporating clinical judgment into longitudinal rare-disease representation learning. It also emphasizes dependence on judgment quality, surrogate approximation, supervision weighting, and clinical validation.

Framework: The framework converts LLM-assigned clinical labels into surrogate supervision that steers a generative representation model.The surrogate provides gradients and makes supervision computationally feasible during model training.
Evaluation: Judgment consistency depended strongly on model size and model family, so LLM quality should be prioritized during offline label generation.The approach was evaluated on spinal muscular atrophy motor-function data using pairwise comparison prompts.
Results: Moderate supervision weights improved consistency between observed and reconstructed judgment distributions while leaving HINE-2 reconstruction errors in a similar range.The added loss therefore did not hinder the numerical reconstruction objective at moderate weights.
Limitations: Synthetic-expert outputs can transmit systematic prompt, labeling, rendering, or LLM errors into the learned latent representation and therefore require careful auditing.Auditing cannot establish that every retained label is clinically correct.
Limitations: The method requires balancing clinical consistency against numerical reconstruction because high supervision weights may reduce reconstruction quality and downstream prediction performance.Low weights may be too weak to affect the representation, while surrogate smoothing can obscure decision boundaries in sparse or ambiguous HINE-2 regions.
Future work: Future work should strengthen clinical validation and propagate synthetic-expert uncertainty into the loss rather than treating uncertainty mainly as a diagnostic property.Possible extensions include down-weighting uncertain labels or using softer target distributions.

Ethics statement

The study used anonymized SMArtCARE registry data reviewed and authorized for research, with all LLM inference conducted locally in secure computing environments to protect confidentiality.

Ethics statement: Anonymized SMArtCARE registry data were reviewed by the University of Freiburg Ethics Committee and authorized by the registry for this research.The ethics review reference was 22-1430-S1-retro.
Ethics statement: All LLM inference was performed locally within secure computing environments to protect participant confidentiality.

Conflicts of interest

Harald Binder reports serving as a Guest Editor for a Biostatistics special collection, while the remaining authors declare no conflicts of interest.

Conflicts of interest: Harald Binder serves as a Guest Editor of a Biostatistics special collection; the remaining authors declare no conflicts of interest.The collection is titled “Statistical Foundations of AI and Real-World Evidence Generation.”

Supplementary materials for: Large language

The supplementary materials accompany a paper titled “Large language models as synthetic clinical experts to inform longitudinal rare-disease modeling.”

The paper title identifies large language models as synthetic clinical experts for longitudinal rare-disease modeling.
The listed authors are Clemens Schächter, Astrid Pechmann, Janbernd Kirschner, and Harald.
The affiliations include the University of Freiburg’s Institute of Medical Biometry and Statistics, Freiburg Center for Data Analysis, Modeling and AI, Department of Neuropediatrics and Muscle Disorders, and CIBSS.

A Data preprocessing and hyperparameters · A.1 Dataset

The analysis used HINE-2 motor-function assessments from children under 12 with spinal muscular atrophy in the SMArtCARE registry. The dataset comprised 13,000 observations from 994 patients, with longitudinal follow-up summarized by visit, age, and duration measures.

A.1 Dataset: The analysis used HINE-2 motor-function assessments from patients under 12 years in the SMArtCARE registry.SMArtCARE tracks motor-function development, motor-milestone achievement, and treatment history in children diagnosed with spinal muscular atrophy.
A.1 Dataset: The HINE-2 profile contained 11 motor-function items spanning upper-limb function, postural control, mobility, and motor milestones.The items covered upper limb function, raising hands, reaching overhead, head control, sitting, voluntary grasp, kicking, rolling, crawling, standing, and walking.
A.1 Dataset: The registry tracks motor-function development, motor-milestone achievement, and treatment history for children diagnosed with spinal muscular atrophy.
A.1 Dataset: 13,000 observations and 994 patients comprised the dataset.The dataset contained exactly 13,000 observations and 994 patients.
A.1 Dataset: 12 visits per patient was the median number of visits.The reported median was 12 visits per patient.
A.1 Dataset: 1.2 years was the median age at first observation, while 3.5 years was the median follow-up duration.

A.2 Synthetic-expert label generation and surrogate training

Synthetic HINE-2 profiles were generated, rendered as age-annotated motor-function descriptions, and evaluated through reversed-order pairwise comparisons among four candidate labels. Valid aggregated labels then supervised a differentiable classifier trained on age and thermometer-encoded HINE-2 profiles.

Synthetic profile generation: 20,000 unique synthetic profiles were retained from a two-dimensional age-conditional VAE, with ages jittered by up to 0.25 years.The synthetic samples supplemented real HINE-2 data and were sampled from the decoder.
Profile rendering: Each profile became an age-annotated textual motor-function description using item-level wording based on HINE-2 descriptions and score-to-text mappings.The resulting item descriptions were concatenated with the patient’s age for prompting.
Synthetic-expert labeling: Four candidate labels—SMA type 1, SMA type 2, SMA type 3, and presymptomatic motor development—were compared pairwise, including label-order reversal.Each profile therefore produced two comparison panels, and the prompt template is provided in Table S2.
Label aggregation and surrogate training: The differentiable surrogate was trained as a classifier to reproduce retained labels from patient age and thermometer-encoded HINE-2 profiles.This training followed aggregation of pairwise answers using the ordinal label structure.

A.3 Hyperparameters · A.4 Milestones and Cox regression

The model used a two-dimensional conditional latent space with specified neural-network, optimization, and synthetic-expert supervision settings. Milestone analyses defined censoring and time-varying Cox intervals, then evaluated one-year landmark risks and calibration.

A.3 Hyperparameters: The encoder and surrogate used single-hidden-layer ReLU networks with 100 units, 41 inputs, Adam optimization, and 50 surrogate-training epochs.Inputs comprised thermometer-encoded HINE-2 profiles plus age; the surrogate used cross-entropy loss on class labels.
A.3 Hyperparameters: The conditional VAE used a two-dimensional age-conditioned latent space, ordinal categorical decoding, Adam at 0.01, β = 0.5, η = 5, and λSE ∈{0, 10, 25, 50, 100}.The listed weights respectively controlled the KL, alignment, and synthetic-expert supervision terms.
A.3 Hyperparameters: The latent mixed-effects model used the same two-dimensional space and fixed effects for onset age, SMN2 copy-number, treatment timing, switching, BMI, height, and age interactions.Treatment-time variables were set to zero before the respective treatment or medication switch.
A.4 Milestones and Cox regression: The SMArtCARE registry milestones were independent sitting, standing, and walking, corresponding to broad motor domains represented by HINE-2 items but using separate milestone definitions.These milestones formed the endpoints for the Cox analysis.
A.4 Milestones and Cox regression: Exact milestone-attainment ages were uncensored events, whereas missing exact times, unknown future attainment, and pre-observation attainment without exact age received right- or left-censoring.Table S3 defines the motor milestones and excludes left-censored patient-milestone pairs and pairs lacking a motor-function visit before the event.
A.4 Milestones and Cox regression: Longitudinal visits were converted into age-scale time-varying Cox intervals spanning observed visits until the next visit, milestone, or censoring time.Evaluated feature sets included HINE-2 total score, clinical covariates, and latent encoder coordinates.
A.4 Milestones and Cox regression: At a fixed one-year landmark horizon, visits were classified by milestone occurrence or event-free status, with inverse-probability-of-censoring and per-patient-visit weighting.Visits censored before the horizon received inverse-probability-of-censoring weighting, and rows were additionally weighted by landmark-visit counts per patient and milestone.
A.4 Milestones and Cox regression: Calibration was summarized by expected calibration error, the weighted mean absolute difference between predicted and observed risk across 10 quantile-based bins.The metric used weighted calibration-bin discrepancies.