PaperlayerView exact revision on arXiv

Source-linked AI summary

PPG-LM: A Photoplethysmography-Language Model with Multi-Level Clinical Alignment

Xiaoda Wang, Minxiao Wang, Maxwell A Xu, Patrick Langer, Kaiqiao Han, Defu Cao, Xiao Luo, Yuzhe Yang, Yan Liu, Xiao Hu, Yizhou Sun, Wei Wang, Carl Yang

arXiv:2609.33516v1cs.AI

TL;DR

PPG foundation models primarily rely on task-specific prediction, while medical knowledge in language models does not necessarily translate into waveform understanding; aligning waveform information with EHR context also requires supervision across temporal scales. PPG-LM pairs PPG with segment-, event-, and visit-level captions and learns alignment in two stages. Experiments across MC-MED, MIMIC-III, and VitalDB support its effectiveness in language-based recognition, retrieval, captioning, and clinical prediction.

  • Problem

    PPG foundation models primarily use task-specific prediction, while textual medical knowledge in LLMs does not ensure understanding of pulse morphology and timing; aligning signal and EHR context also requires matching local, event, and visit timescales.

    Source: §1 · pp. 1–2

  • Method

    PPG-LM pairs signal- and EHR-derived captions at segment, event, and visit levels, then learns segment correspondence before extending alignment to events and visits.

    Source: §1 · p. 2

  • Results

    Experiments across MC-MED, MIMIC-III, and VitalDB support PPG-LM’s effectiveness in language-based recognition, retrieval, captioning, and clinical prediction.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The experiments support connecting PPG waveforms with clinical context across segment, event, and visit levels.

    Source: §6 · p. 10

    PPG-LM is intended for research and has not been validated for clinical decision-making; medication-event predictions reflect associations, not causal treatment effects.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Photoplethysmography (PPG) is widely recorded by clinical monitors and consumer wearables, providing a scalable source of continuous physiological information. These recordings offer an opportunity for physiological assessment at scale, but realizing this potential requires models to learn from both signal-derived physiological supervision and broader clinical context captured in electronic health records (EHRs). This involves aligning information spanning local observations, care events, and entire visits with PPG representations at corresponding temporal scales. However, existing PPG foundation models primarily rely on task-specific prediction heads, while the medical knowledge of large language models does not necessarily translate into waveform understanding. To bridge this gap, we introduce PPG-LM, the first PPG-language model family to learn physiological representations from both signal-derived supervision and broader clinical context captured in EHRs. To construct clinically grounded captions, we develop an automatic captioning pipeline that generates segment-, event-, and visit-level descriptions from signal measurements and structured EHR records. We then learn from these pairs through a two-stage framework that first establishes segment-language correspondence through contrastive learning and waveform-conditioned captioning, then extends alignment to events and visits through time-aware aggregation and temporal statement matching. Pretrained on approximately 73k hours of PPG, PPG-LM supports language-based recognition, cross-modal retrieval, and segment captioning. Experiments on MC-MED, MIMIC-III, and VitalDB show improved retrieval and caption factuality over language-model baselines and gains over PPG and time-series foundation models on multiple clinical prediction tasks.

1 INTRODUCTION

PPG models and language models each leave a gap in connecting waveform properties with clinical context. PPG-LM uses multi-level captions and two-stage alignment to address that gap, and is evaluated across three datasets and several clinical tasks.

1 INTRODUCTION

  • PPG-LM uses shared waveform and text encoders: Stage 1 establishes segment–language correspondence, then Stage 2 extends alignment to medication events and variable-length visits.Stage 1 combines contrastive learning and waveform-conditioned captioning; Stage 2 uses signed time offsets, visit latents, and temporal statement matching.Source: §1 · p. 2
  • PPG-LM is the first PPG–language model family described here to learn physiological representations from multi-level clinical language supervision, training on approximately 73,700 hours of PPG.It is designed to support recognition, retrieval, and captioning.Source: §1 · p. 2
  • An automatic captioning pipeline combines signal measurements and structured EHR records into segment-, event-, and visit-level descriptions while preserving each description’s evidence source and temporal scope.Source: §1 · p. 2
  • Experiments on MC-MED, MIMIC-III, and VitalDB assess recognition, retrieval, captioning, temporal understanding, and clinical prediction against PPG FMs and LLMs, demonstrating language capabilities and transferable representations.Source: §1 · p. 2

2 RELATED WORK

Prior PPG and time-series models generally leave waveform–language correspondence unmodeled or address only particular estimates and reports. Existing alignment work does not jointly ground PPG descriptions across local physiology, care events, and visit-level context.

Foundation models for PPG and time series

  • PPG foundation models use contrastive, non-contrastive, morphology-aware, generative, and cross-signal objectives, but their signal-centered training leaves waveform–language correspondence unmodeled and clinical applications typically need task-specific predictors.Source: §2 · p. 3

LLMs for PPG modeling

  • PPG LLM methods focus on prescribed estimates, question answering, or report generation, leaving joint waveform–language learning across clinical temporal levels unaddressed [Pham et al., 2026].Source: §2 · p. 3

Time series and language alignment

  • Time-series–language alignment spans wearable, clinical-signal, and multilevel medical-imaging work, but does not directly align PPG across local physiology, care events, and visit-level clinical information [Chen et al., 2026].Source: §2 · p. 3

3 CLINICALLY GROUNDED CAPTION GENERATION

The captioning pipeline pairs PPG with descriptions generated from signal measurements and structured EHRs at segment, medication-event, and visit scales. Each scale captures a different scope of physiological or clinical information.

3 CLINICALLY GROUNDED CAPTION GENERATION

  • The training corpus uses MC-MED emergency-department PPG and structured EHRs, generating captions paired with observations at corresponding temporal scales.Captions use fixed language templates with reproducible selection and sentence shuffling.Source: §3 · p. 3, Figure 2 · p. 3
  • Segment captions describe up to eight PPG-window attributes, including pulse rate, rhythm regularity, signal quality, morphology, and beat-to-beat amplitude variability.Available fields yield numerical or qualitative descriptions.Source: §3 · p. 3
  • Event captions name an eligible medication class and pair it with up to three latest pre-administration windows and up to five evenly spaced post-administration windows.Controls use the same sampling rule while excluding anchors near recorded administrations.Source: §3 · p. 4
  • Visit captions combine encounter records and demographics with summarized prior diagnoses and active home medications, pairing the resulting description with observations across the encounter.Ordered segment rates and rhythm labels also yield temporal statements about state occurrence, proportion, duration, and rate ordering.Source: §3 · p. 4

4 METHOD

PPG-LM uses shared waveform and text encoders to align PPG with captions at segment, event, and visit scales. Its two stages move from segment–language learning to temporal aggregation and matching across broader clinical scopes.

4 METHOD

  • Shared waveform and text encoders reuse features across description levels, with scope-specific read-outs for segment, event, and visit alignment.The waveform encoder maps each PPG window to a feature; text embeddings use a learned prefix and level-specific projection.Source: §4 · pp. 4–5

4.1 STAGE 1: SEGMENT–LANGUAGE PRETRAINING

  • Stage 1 combines symmetric contrastive alignment of PPG windows and captions with waveform-conditioned token prediction, and retains this segment objective in Stage 2.A separate attention pooler forms waveform memory for the caption decoder; the combined objective trains both shared encoders from scratch.Source: §4.1 · p. 5

4.2 STAGE 2: MULTI-LEVEL ALIGNMENT

  • Stage 2 aggregates shared window features for event and visit descriptions, adds temporal-statement supervision, and retains segment alignment.It initializes from Stage 1 and alternates segment, visit, and event batches.Source: §4 · p. 4, §4.2 · pp. 5–6
  • Event alignment pools nearby PPG windows with signed time offsets relative to a medication or control anchor, distinguishing before- and after-anchor observations.Candidate sets exclude other views of the same class to avoid contrasting identical treatment categories.Source: §4.2 · p. 5
  • Visit alignment selects up to 256 windows, encodes monitoring and arrival times with gap and evidence tokens, then uses eight latents for global alignment and temporal-statement matching.Each statement can select its most compatible latent; training ranks the true statement set above a version with one incompatible statement, masking missing statements and unavailable corruptions.Source: §4.2 · pp. 5–6

5 EXPERIMENTS

Across MC-MED, MIMIC-III, and VitalDB, the experiments assess language-based physiological recognition, retrieval, captioning, event- and visit-level recognition, and clinical prediction. Results span zero-shot evaluations, frozen-representation probes, and model analyses.

5 EXPERIMENTS

  • Experiments use MC-MED for pretraining and in-domain evaluation, with MIMIC-III and VitalDB for external evaluation, and compare against language, PPG, and time-series model baselines.The frozen-representation evaluation uses patient-disjoint splits and the same four windows per visit across models.Source: §5 · p. 6, §5.3 · p. 8

5.1 ZERO-SHOT PHYSIOLOGICAL UNDERSTANDING

  • PPG-LM achieves the highest scores on 10 of 11 zero-shot physiological-recognition pairs; for estimation, it leads in fact accuracy on all three datasets and has the lowest reported errors for displayed measurements.VitalDB respiratory-rate recognition is the exception (0.329 vs. 0.330 for Opus 5.5); heart-rate MAE is 3.90–4.27 bpm against an independent ECG reference.Source: Table 2 · p. 6, §5.1 · p. 7, Table 3 · p. 7

5.2 ZERO-SHOT EVENT- AND VISIT-LEVEL PHYSIOLOGICAL RECOGNITION

  • PPG-LM outperforms reported baselines on all displayed medication-class AUROC comparisons and leads on seven of eight visit-level tasks.Visit-level balanced accuracy is 0.520–0.570, indicating modest discrimination; the exception is hypertension (0.520 vs. 0.523 for GPT-6-sol).Source: §5.2 · pp. 7–8, Table 5 · p. 8, Table 6 · p. 8

5.3 CLINICAL PREDICTION WITH FROZEN REPRESENTATIONS

  • Stage 2 PPG-LM exceeds all six foundation-model baselines on 32 clinical prediction tasks and Stage 1 on 30, with VitalDB hypertension and EtCO2 as exceptions.Glucose prediction remains weak (R2 = 0.083).Source: §5.3 · p. 8, Table 7 · p. 9

5.4 MODEL ANALYSIS

  • Ablations indicate that temporal matching supports temporal discrimination, event offsets support event discrimination, and continued segment supervision preserves retrieval.CoCa gives the best Stage-2 rhythm and quality recognition and retrieval among the compared CLIP and SigLIP variants.Source: §5.4 · p. 8, Table 8 · p. 8
  • Mean HR-class macro-F1 rises consistently as labeled examples per class increase from 5 to 50 across MC-MED, MIMIC-III, and VitalDB.The probes use frozen PPG-LM representations.Source: §5.4 · p. 8, Figure 4 · p. 9
  • During extended Stage-2 CoCa training, segment retrieval improves while medication-event AUROC largely plateaus by 50–60k steps.The two tasks therefore show different benefits from additional optimization.Source: §5.4 · p. 9, Figure 4 · p. 9
  • Visit retrieval improves when context grows from four to sixteen windows, with little further gain from longer contexts.This result is for the fixed Stage-2 CoCa model.Source: §5.4 · p. 9, Figure 4 · p. 9

6 CONCLUSION

The paper presents PPG-LM as a model family connecting PPG waveforms with clinical context through multi-level captions and two-stage alignment. Experiments support its effectiveness across language tasks and clinical prediction, while the study notes limits on clinical use and describes its use of generative AI for language polishing.

6 CONCLUSION

  • PPG-LM combines signal- and EHR-derived captions with two-stage alignment across segment, event, and visit levels.Experiments on MC-MED, MIMIC-III, and VitalDB support effectiveness in language-based recognition, retrieval, captioning, and clinical prediction.Source: §6 · p. 10
  • Generative AI tools were used for language polishing, and the authors reviewed edits and retained final editorial control and responsibility.Source: AI USE STATEMENT · p. 10
  • The model is intended for research, not clinical decision-making; prospective validation and subgroup and generalizability assessment are needed before deployment.Medication-event predictions reflect associations, not causal treatment effects.Source: ETHICS STATEMENT · p. 10
  • Appendices provide captioning rules, templates, evaluation prompts, architecture and training details, and dataset and evaluation protocols.Source: REPRODUCIBILITY STATEMENT · p. 10

A LIMITATIONS

The paper identifies data, supervision, and validation boundaries for PPG-LM, then details how segment, visit, event, and temporal captions are constructed. The captioning procedures impose specific selection and timing rules, while event comparisons do not establish causal effects or remove all temporal confounding.

A LIMITATIONS

  • Pretraining uses single-center adult emergency-department data; EHR completeness and temporal accuracy may limit caption quality and generalization.Broader multicenter data, richer longitudinal records, clinician-verified descriptions, prospective validation, subgroup analyses, and uncertainty calibration are identified as needed before clinical deployment.Source: A LIMITATIONS · p. 15

B CAPTION CONSTRUCTION DETAILS

  • Caption construction pairs signal measurements and clinical records with PPG at segment, event, visit, and temporal scopes.Tables summarize caption content, with language templates and rendered examples in Appendix E.Source: B CAPTION CONSTRUCTION DETAILS · p. 15

B.1 SEGMENT CAPTIONS AND SIGNAL QUALITY

  • Segment descriptions use 30-second, 125-Hz PPG windows and up to eight attributes; poor-quality windows are excluded from training but retained for validation and testing.Optional quality checks may be absent without rejecting a window, and unavailable or unreliable ECG rhythm annotations are omitted.Source: §B.1 · p. 15, Table 9 · p. 15

B.2 VISIT CAPTIONS AND CLINICAL CONTEXT

  • Visit captions summarize encounter context, demographics, history, medications, and temporal statements, but some fields have incomplete evidence of when they were available.Known post-arrival diagnoses, disposition, discharge diagnoses, and length of stay are excluded; home medications lack encounter-specific temporal verification.Source: §B.2 · p. 16, §B.4 · p. 16, Table 10 · p. 16

B.3 EVENT CAPTIONS AND WITHIN-VISIT CONTROLS

  • Event captions map eligible medication administrations to drug classes and compare nearby pre- and post-administration windows with controls from the same monitored visit.The sampling avoids a fixed window-start cue but does not establish causal effects or remove all temporal confounding.Source: §B.3 · p. 16, Table 11 · p. 16

B.4 TEMPORAL DESCRIPTIONS AND CORRUPTED STATEMENTS

  • Temporal descriptions derive from ordered segment-caption rates and rhythm labels; unknown states and monitoring gaps interrupt runs, and unsupported or ambiguous statements are omitted.Corrupted descriptions replace one supported operator value with an incompatible value while retaining other sentence facts.Source: §B.4 · pp. 16–17, Table 12 · p. 17

C METHOD AND TRAINING DETAILS

The appendix details PPG-LM’s shared waveform and text encoders, segment-level contrastive and captioning objectives, and time-aware event and visit aggregation. A two-stage schedule preserves segment alignment while adding broader clinical supervision.

C.1 WAVEFORM AND TEXT ENCODERS

  • The waveform encoder processes standardized 30-second windows as 150 patches, while a shared causal text tower uses level-specific prefixes and projections.Both towers start from random initialization; CLIP provides the tokenizer and contrastive formulation, not pretrained encoder weights.Source: §C.1 · p. 18

C.2 COCA ALIGNMENT AND CAPTIONING OBJECTIVES

  • CoCa alignment pairs a single-query waveform read-out with a text end-of-sequence embedding, while a waveform-conditioned decoder supplies segment caption supervision.Symmetric InfoNCE masks off-diagonal pairs with equal content keys; caption loss updates the shared waveform and text representations.Source: §C.2 · pp. 18–19

C.3 EVENT AND VISIT AGGREGATION

  • Event aggregation combines windows around medication or control anchors using signed time offsets, while visit aggregation represents selected windows, monitoring gaps, and coverage evidence with eight learned latents.Visit input can include up to 256 windows, 16 gap tokens, and an evidence token; global and temporal latents support clinical-caption alignment and temporal matching.Source: §C.3 · pp. 19–20

C.4 PRESERVING SEGMENT ALIGNMENT

  • Stage 2 retains segment caption training and uses a frozen Stage-1 waveform encoder as a reference to limit changes to normalized segment embeddings.The distillation weight is 0.1; the penalty constrains embedding directions while allowing pre-projection waveform features to adapt.Source: §C.4 · pp. 20–21

C.5 OPTIMIZATION, FEATURE CACHING, AND INFERENCE

  • Stage 1 trains from scratch for six epochs on 8,842,271 windows; Stage 2 runs 30,000 updates, cycling segment, visit, and event batches at a 4:2:1 ratio.Visit batches use cached context with up to 256 windows per visit and at most eight live windows; inference does not require the training cache.Source: §C.5 · pp. 21–22
  • The appendix specifies a two-stage optimization schedule and reports approximately 26.3 hours for Stage 1 and 6.0 hours for Stage 2 on one NVIDIA H200 GPU.Both stages use AdamW; Stage 2 initially freezes the waveform and text towers and segment logit scale for 2,000 updates.Source: §C.5 · pp. 21–22

D.1 EXPERIMENTAL SETUP

The evaluation covers emergency, intensive-care, and intraoperative monitoring, with MC-MED used for pretraining and MIMIC-III and VitalDB for external transfer. As retrieval pools grow, recall falls, especially on external datasets.

D.1 EXPERIMENTAL SETUP

  • MC-MED supplies pretraining data, while MIMIC-III and VitalDB evaluate transfer to external intensive-care and intraoperative settings.Clinical probes fit on target-dataset training patients and evaluate on patient-disjoint test patients; language benchmarks use fixed items and class definitions.Source: §D.1 · pp. 22–23

D.2 CLINICAL TASK DEFINITIONS

  • The benchmarks distinguish waveform physiology, medication events, visit facts, and clinical prediction targets, with retrieval testing whether paired signals and descriptions rank together.Medication-event AUROC compares one administration class against no-administration controls; recorded administration does not assert a causal physiological response.Source: §D.2 · pp. 24–25

D.3 CROSS-MODAL RETRIEVAL

  • At 2,000 candidates, MC-MED S2T/T2S R@1 is 0.780/0.744 and R@10 is 0.998/0.995; MIMIC-III and VitalDB R@10 is 0.853/0.762 and 0.510/0.483, respectively.Expanding from 100 to 2,000 candidates lowers recall in both directions, with larger R@1 declines on the external datasets.Source: §D.3 · p. 27, Table 18 · p. 28

D.4 CAPTION EVALUATION AND NUMERICAL READ-OUT

Caption evaluation finds strong parser-based factual accuracy for generated and retrieved descriptions, while judge-based ratings favor retrieval. Generated-caption heart-rate read-out performs best against ECG references, and PPG-LM also leads the reported MC-MED visit-fact tasks.

D.4 CAPTION EVALUATION AND NUMERICAL READ-OUT

  • On MC-MED, eight-field fact accuracy is 0.932 for direct generation and 0.889 for nearest-caption retrieval, versus 0.923 and 0.877 on the common six fields.A stated HR is correct within ±5 bpm; fact accuracy counts correct stated fields, while coverage is the fraction of gradable fields stated.Source: §D.4 · p. 28, Table 19 · p. 29
  • Generated PPG-LM captions are preferred to GPT-5.6-luna captions at rates of 0.922, 0.855, and 0.777 on MC-MED, MIMIC-III, and VitalDB, respectively.Direct generation has higher parser fact accuracy on all three datasets, but nearest-caption retrieval receives higher judge factuality scores and fewer unsupported claims.Source: §D.4 · p. 28
  • Generated-caption HR read-out has the lowest MAE and highest fraction within 5 bpm on all three datasets; on MC-MED, its MAE is 3.90 bpm against ECG.Its MAE against the PPG-derived annotation rate is 0.27 bpm; ECG provides an independent reference beyond the caption-construction target.Source: §D.4 · p. 29, Table 20 · p. 29

D.5 ADDITIONAL VISIT-LEVEL CLINICAL-FACT RECOGNITION

  • PPG-LM achieves the highest reported balanced accuracy on all 19 MC-MED visit-fact tasks, reaching 0.785 for HR and 0.729 for fever, with medical-history scores from 0.580 to 0.687.These scores measure associations with recorded visit attributes rather than independent clinical diagnoses.Source: §D.5 · p. 30, Table 21 · p. 30

D.6 MEDICATION-EVENT EVALUATION

The medication-event evaluation uses class-specific administration-versus-control comparisons. PPG-LM’s balanced-accuracy comparison with the LLM baseline differs by dataset.

D.6 MEDICATION-EVENT EVALUATION

  • The main benchmark compares each medication class with no-administration controls, excluding other medication classes from that comparison’s negative group.It uses 393 administrations and 393 controls on MC-MED, and 139 of each on VitalDB.Source: §D.6 · p. 30
  • The supplementary event cohort includes 150 administrations and 150 controls on MC-MED and 60 of each on VitalDB; MIMIC-III has no event source in that benchmark.Same-visit paired accuracy uses only 56 MC-MED and 19 VitalDB pairs with both views available.Source: §D.6 · p. 30
  • Using a continuous difference between highest medication-class and no-medication scores, PPG-LM achieved balanced accuracy 0.647 versus LLM 0.550 on MC-MED, and 0.533 versus 0.633 on VitalDB.Source: §D.6 · p. 31

D.7 FULL CLINICAL PREDICTION RESULTS

Stage 2 outperformed all six foundation-model baselines across 32 clinical prediction tasks and improved over Stage 1 on 30, with limited predictive performance for several targets. In few-shot adaptation, additional labels generally improved performance, although gains varied by task.

D.7 FULL CLINICAL PREDICTION RESULTS

  • Clinical prediction probes use frozen representations and dataset-specific patient splits; their targets and laboratory timing differ, so they assess association with recorded labels rather than a uniform prospective endpoint.Visit-level comparisons use the same four PPG windows across models, but Stage 2 uses a learned visit aggregator while baselines and Stage 1 mean-pool features.Source: §D.7 · p. 31

D.8 FEW-SHOT ADAPTATION

  • At k = 50, HR-class macro-F1 reached 0.921, 0.912, and 0.873 on MC-MED, MIMIC-III, and VitalDB, respectively; 11 of 12 tasks peaked at k = 50.VitalDB ICU admission instead peaked at k = 10 (0.591) and fell to 0.557 at k = 50.Source: §D.8 · p. 31, Table 23 · p. 33
  • The PulseLM comparison uses a distinct in-context protocol with ten labeled windows total, so its results should be interpreted under that support budget rather than k examples per class.On MC-MED, PulseLM’s linear probes at k = 10/50 reached rhythm macro-F1 of 0.395/0.403 and HR-class macro-F1 of 0.750/0.811.Source: §D.8 · p. 31

D.7 FULL CLINICAL PREDICTION RESULTS

  • Stage 2 exceeded all six foundation-model baselines on all 32 tasks and improved over Stage 1 on 30, while low R2 indicated limited performance for electrolytes, glucose, and BMI.Stage 1 was slightly stronger on VitalDB hypertension (0.656 vs. 0.655) and EtCO2 (0.153 vs. 0.152).Source: §D.7 · p. 31, Table 22 · p. 32

D.9 OBJECTIVE ABLATIONS

The objective and prompt ablations show that segment training objectives and class-description choices affect recognition and retrieval results. The appendices document how captions and prompts are constructed and how the waveform-conditioned decoder uses them.

D.9 OBJECTIVE ABLATIONS

  • At Stage 2, CoCa improved all three reported metrics, led rhythm and quality macro-F1, and matched SigLIP’s R@1, while retaining caption generation.Its signal-only-caption R@1 was 0.820, versus 0.780 for the main signal-plus-vitals configuration at the same candidate-pool size.Source: Table 24 · p. 34, §D.9 · p. 34

D.10 PROMPT SENSITIVITY

  • Template prompts performed better on seven of nine dataset–task pairs; adding reworded descriptions improved HR-class macro-F1 on MC-MED and MIMIC-III but reduced it on VitalDB.Template prompts achieved higher rhythm and quality scores on all three datasets.Source: Table 25 · p. 34, §D.10 · p. 34

E LANGUAGE TEMPLATES AND EXAMPLES

  • Training captions are rendered from structured measurements and clinical records using fixed template banks, with the example identifier and random seed determining template selection and sentence order.No external LLM is used; training targets are distinct from zero-shot class descriptions and learned text-prefix embeddings.Source: E LANGUAGE TEMPLATES AND EXAMPLES · p. 35

E.1 REPRESENTATIVE CAPTION TEMPLATES

  • Segment templates describe available signal attributes, event templates identify medication classes, and visit templates combine available demographics, triage context, and prior history.Temporal templates encode occurrence, prevalence, contiguous duration, or order; negatives replace one eligible sentence while retaining the others.Source: §E.1 · p. 35, Table 26 · p. 35

E.2 ZERO-SHOT EVALUATION PROMPTS

  • Zero-shot segment recognition averages embeddings for two caption-style descriptions per class and scores them against waveform embeddings by cosine similarity; visit and event prompts use selected read-outs.The event bank includes ten medication classes and a no-medication description; the main medication table reports separate class-versus-control AUROCs.Source: §E.2 · p. 35, Table 27 · p. 36

E.3 SYNTHETIC CAPTION EXAMPLES AND DECODER CONTEXT

  • Synthetic examples illustrate training targets rather than patient records or model outputs, while the CoCa decoder cross-attends to waveform memory to predict caption tokens.Missing fields contribute no sentences; the shared text encoder prepends a learned segment prefix before causal text states enter the separate multimodal decoder.Source: §E.3 · p. 36

Related papers

Loading 2609.33516v1…