PaperlayerView exact revision on arXiv

Source-linked AI summary

PhenoBench: Mapping What a Deeply Phenotyped Human Cohort Can Tell Us

Gal Sapir, Alon Diament, Adva Wolf, Doron Yaya-Stupp, Dikla Gelbard Solodkin, Dana Azouri, Anat Etzion-Fuchs, Guy Lutsker, Eran Segal, Hagai Rossman

arXiv:2609.06080v2cs.LGcs.AI

TL;DR

Deeply phenotyped cohort analyses can be difficult to compare because studies use different populations, inputs, splits, baselines, and methods. PhenoBench turns HPP measurements into explicit tasks and controlled comparisons. Pretrained tabular models had small average gains over ridge, while language models generally did not outperform task-specific fitted models.

  • Problem

    Studies of deeply phenotyped cohorts use heterogeneous populations, inputs, splits, baselines, and methods, making their findings difficult to compare directly.

    Source: §1 · pp. 1–2

  • Method

    PhenoBench translates HPP measurements into clinically grounded tasks with fixed evaluation conditions for comparing information sources and predictive models.

    Source: §1 · p. 2, §2.1 · p. 2

  • Results

    Pretrained tabular models ranked higher overall but improved only slightly over ridge under the bounded protocol, while task-specific fitted models generally outperformed language models.

    Source: §4 · p. 8

  • Takeaways & Limitations

    A versioned evidence record retains each result’s question, conditions, and provenance so new tasks and models can be added without redefining earlier comparisons.

    Source: §4 · p. 8

    The tabular comparison excludes recommended ensembles and exhaustive library-specific tuning, so small gains do not settle either model family’s best attainable performance.

    Source: §4.1 · p. 9

Abstract

from arXiv · show

Deeply phenotyped cohorts combine clinical, imaging, molecular, and wearable observations across timescales from seconds to years, but heterogeneous analyses are not directly comparable. We present PhenoBench, an executable benchmark that turns deep-phenotyping measurements into explicit questions and controlled comparisons of information sources and predictive models. It is built around the Human Phenotype Project, with more than 13,000 participants at the initial visit. Each question fixes the target, population, timing, and allowed information; its evaluation contract specifies the split, metric, baseline, and claim boundary. PhenoBench defines 90 clinically grounded tasks across 15 domains and 26 input modalities. Across 160 matched regression comparisons spanning 52 tasks, six pretrained tabular models ranked above the evaluated task-specific baselines, including XGBoost and CatBoost, under a fixed single-estimator protocol with bounded tuning. Giving each task equal weight, their mean advantage over ridge was 0.0103 $R^2$ (95% task-bootstrap interval, 0.0071-0.0136). We also evaluated 14 language models, collectively covering 40 tasks spanning phenotype recovery, classification, follow-up forecasting, and participant ordering. Without cohort-specific fitting, language models made informative predictions on some tasks but showed task-specific capability gaps, shared failures of scale, and rarely surpassed task-specific ridge or logistic regression models fitted on the same input fields. PhenoBench provides a versioned, auditable evaluation system where new questions, measurements, and models can be added without redefining existing comparisons.

1. Introduction

Deeply phenotyped cohorts offer measurements across health domains and timescales, but studies using different populations, inputs, splits, baselines, and methods cannot be directly compared. PhenoBench addresses this by turning HPP measurements into clinically grounded questions with explicit evaluation contracts.

1. Introduction

  • HPP combines clinical, imaging, molecular, and dense physiological measurements with follow-up over years, but individual studies use conditions that make their findings difficult to compare.More than 13,000 participants had completed the initial visit.Source: §1 · p. 1
  • Deep-phenotyping measurements do not themselves specify prediction tasks; targets, eligible populations, temporal relations, and relevant baselines must be defined.This makes task definition a distinct benchmarking problem from defining tasks over encounter-based EHR data.Source: §1 · pp. 1–2
  • Evaluating predictive models in health requires fixing the question, eligible population, available information and timing, relevant baseline, and success metric.These choices encode clinical judgment and determine which comparisons are meaningful and what claims their results support.Source: §1 · p. 2
  • PhenoBench provides 90 clinically grounded tasks across 15 domains and 26 modalities, with contracts for controlled comparisons of measurements and methods.Its evaluations include 160 matched tabular-model comparisons spanning 52 tasks and language-model evaluation covering 40 tasks.Source: §1 · p. 2

2. Results

PhenoBench organizes HPP measurements into fixed, clinically grounded tasks and compares information sources and models under controlled conditions. Results show question-dependent measurement value, small average advantages for pretrained tabular models over ridge, and task-specific gaps among language models.

2.1. Mapping clinically grounded questions across HPP

  • PhenoBench defines 90 tasks across 15 clinical domains and 26 input modalities, spanning prediction, retrieval, matching, and causal effect estimation over timescales from seconds to years.Participant-level evaluation cohorts range from 537 to 11,726 people.Source: §2.1 · p. 2, Figure 1 · p. 3

2.2. Mapping predictive information across tasks

  • Which measurement appeared most informative depended on the question: CGM had the largest validation-set ΔR^2 point estimate for fasting glucose, while WatchPAT did for chronological age.The question-first comparisons hold the question and evaluation conditions fixed while changing the measurement; small and negative ΔR^2 values are retained, with negative values indicating reduced performance versus the matched baseline.Source: §2.2 · pp. 2, 4, Figure 2 · p. 4

2.3. Comparing model families on matched tasks

  • The six pretrained tabular models’ mean advantage over ridge was 0.0103 R^2 (95% task-bootstrap interval, 0.0071–0.0136) across 52 equally weighted tasks.The advantage was positive in 43 tasks; results describe the evaluated single-estimator protocol, and better aggregate rank did not imply large absolute gains across all comparisons.Source: §2.3 · p. 4
  • Frozen Chronos-2 embeddings underperformed engineered features from the same signal in 25 of 31 CGM comparisons and 30 of 32 exact-intersection WatchPAT comparisons.Where glucose embeddings performed better, gains were concentrated in three metabolic targets; the gap narrowed with larger training cohorts.Source: §2.3 · pp. 4–5

2.4. PhenoBench-LLM: cohort data as a controlled test of LLM capability

  • On the 11 tasks shared by all 14 language models, Llama 4 Maverick, GPT-5.6, and Gemini 3.7 Flash formed the win-rate-versus-cost Pareto frontier.The shared set included six ordering, three classification, and two phenotype-recovery tasks, but no follow-up forecasting tasks.Source: §2.4 · p. 5
  • Task-specific fitted models generally outperformed language models across phenotype recovery, classification, and ordering; language models exceeded same-field ridge probes in only 8 of 304 paired model–task comparisons.For follow-up tasks, language models received each participant’s baseline measurement; relative to carrying that value forward, their gains were inconsistent, and ridge models using the same inputs performed better.Source: §2.4 · p. 5

3. Methods

The methods define participant-level cohorts and splits, then compare measurement representations and model families under task-specific evaluation contracts. Cross-task summaries weight tasks equally and use task-level resampling to account for multiple comparison cells per task.

3.1. Study design

  • HPP analyses use task-specific eligible subsets and participant-grouped 70/15/15 train, validation, and test splits, with preprocessing fit on training data only.Representation comparisons use the exact participant cohort shared by both representations.Source: §3.1 · p. 5

3.2. Evaluations

  • Each model-comparison cell fixes the task, cohort, predictor, split, target, participants, and feature set while varying the fitting method against a shared demographics-only ridge baseline.The regression capacity analysis compared ten methods across 160 complete test-set cells spanning 52 tasks.Source: §3.2 · pp. 5–6
  • Evaluations compared frozen zero-shot Chronos-2 embeddings with engineered features from the same signal under a ridge head, and language-model predictions with demographic and fitted-probe references.Language-model evaluations used fixed evidence packets and task contracts; schema-faithful synthetic bundles expose the external-model input and output contract without releasing participant data.Source: §3.2 · p. 7

3.3. Analysis

  • The primary tabular cross-task summary averaged cells within each task, gave each of 52 tasks equal weight, and bootstrapped whole tasks to estimate the pretrained-model advantage over ridge.Cell-level tests were descriptive; sensitivity analyses included task-level pairwise tests and equal-model-family weighting.Source: §3.3 · p. 7

4. Discussion

PhenoBench makes measurement–question and model comparisons interpretable through fixed evaluation conditions, while supporting a versioned evidence record that can grow without redefining earlier comparisons. The authors caution that findings are bounded by the evaluated protocol, HPP coverage, and validation setup.

4. Discussion

  • Fixing the clinical question and evaluation conditions lets PhenoBench compare how measurements and models use HPP’s breadth, while new tasks and models can extend a versioned record.Each result retains its question, conditions, and provenance so later studies can distinguish stronger evidence from a different method or question.Source: §4 · p. 8
  • Across evaluated task subsets, informative measurements depended on the question and representation, and pretrained tabular models ranked higher overall but improved only slightly over ridge under the bounded protocol.Engineered glucose and sleep features often outperformed Chronos-2 embeddings of the same signals.Source: §4 · p. 8
  • Across four task categories, language models had distinct capability profiles, cost did not determine performance, and task-specific fitted models generally performed better.Fixed participants, evidence, targets, and metrics exposed shared failures of scale and target semantics.Source: §4 · p. 8

4.1. Limitations

  • Generalizability to other cohorts remains untested, and tabular results do not establish either model family’s best attainable performance because the protocol excludes recommended ensembles and exhaustive library-specific tuning.The authors also caution that rankings may reflect HPP recruitment and measurement coverage, predictive associations may be mistaken for causal or clinically actionable evidence, and uncontrolled data release could compromise privacy.Source: §4.1 · p. 9

5. Public benchmark leaderboards

The compact leaderboards summarize matched tabular-model cells and a common-task language-model comparison. The language-model leaderboard reports pairwise win rates across 11 tasks evaluated by all 14 models.

6. Public benchmark leaderboards

  • The compact model-family leaderboard reports mean within-cell rank across 160 matched cells, where lower rank is better.Methods are compared only within cells sharing the task, cohort, feature set, and split.Source: §5 · p. 9
  • The language-model leaderboard reports pairwise win rates on 11 tasks evaluated by all 14 models, with task-bootstrap intervals.Native metrics are compared only within a task, and the validation-set summary is descriptive.Source: §5 · p. 9
  • In the reported common-task results, Gemini 3.7 Flash had an 82.5% win rate (67.1–94.4%), compared with Claude Opus 4.8 at 73.4% (63.6–82.5%).Source: §5 · p. 9
  • The listed results report GPT-5.4 nano at 20.6%, with an accompanying 8.0–35.7% range; nearby entries include Pixtral Large at 25.9% and DeepSeek V4 Pro at 30.1%.Source: §5 · p. 10

6. Competing interests

The authors report employment and paid-consulting relationships with Pheno.AI Ltd.

6. Competing interests

  • Seven authors are Pheno.AI Ltd. employees, and Eran Segal is a paid consultant to the company.Source: §6 · p. 10

A. Task curation and ingestion audit

The ingestion audit describes a publication-led workflow that moved from candidate questions and task structures through executable contracts to real-data review. Task Cards and a staged contribution path formalize scientific meaning, implementation choices, comparators, provenance, and claim boundaries.

A. Task curation and ingestion audit

  • Publication ingestion captures the question’s target, cohort, timing, information source, metric, comparison, and result, then adds clinical context and a human-readable Task Card.A clinician-authored surrogate-target inventory and later benchmark work also seeded questions beyond those represented in reviewed publications.Source: A. Task curation and ingestion audit · pp. 11–12
  • The 61-study corpus pass assigned studies to four queues: 13 executable with existing components, 11 needing an artifact, 15 needing benchmark capability, 6 needing manual design, and 16 rejected or deferred.The queue distinctions reflected whether existing patterns sufficed, an artifact or reusable capability was missing, target or claim semantics needed author judgment, or evidence and coverage were inadequate.Source: A. Task curation and ingestion audit · p. 12
  • Each applied row required a real-data result, a relevant baseline, a comparability label, and an explicit statement of what could not be claimed.After real-run checks, one study yielded a close paper-inspired comparison, ten useful but non-parity analogues, and two weak rows; runnable configurations alone were insufficient.Source: A. Task curation and ingestion audit · p. 12
  • Prior studies can expand an existing target’s information sources or require different contracts, such as event-level temporal ordering or versioned model-output artifacts.Examples include VAT questions reusing an existing target, a sleep-derived Predictor for resting heart rate, and age-model outputs retained as versioned artifacts.Source: A. Task curation and ingestion audit · p. 12

A.1. Task Card evolution and contribution path

  • Task Cards were added after the earliest executable Tasks and later backfilled, so the current card-first path is prospective rather than a claim about every historical Task.Source: §A.1 · p. 12
  • Task Cards record question and target, HPP measurement binding, clinical context, evaluation plan, risks and limits, and provenance.The separation of task code, configurations, strategies, run artifacts, and Task Cards keeps enforceable scoring facts distinct from scientific meaning and model assumptions.Source: §A.1 · p. 13
  • The contribution path binds a documented target and its cohort, timing, quality control, exclusions, and evaluation grain before deciding whether a new Task or an extension is needed.It then creates or revises a Task Card, fixes comparators before held-out results, implements and tests an executable contract, and records real-data interpretation and unresolved questions.Source: §A.1 · p. 13

A.2. Extended results: Mapping information value across tasks

The extended results map information value across a diverse task collection and show that predictive gains depend on the question, cohort, and representation. The accompanying model analysis defines matched-cell comparisons and documents model configurations and exclusions.

A.2. Extended results: Mapping information value across tasks

  • The registered collection covers 90 tasks across 15 domains and includes regression, classification, sequence, causal-effect, and ranking targets; three framework-only diagnostics are excluded.The supplied figure caption’s stacked segments encode target type, and bar length counts registered tasks rather than evaluated performance.Source: §A.2 · p. 13, Figure 1 · p. 14
  • 0.195 ΔR2 for fasting glucose, 0.091 for LDL cholesterol, and 0.018 for gut microbial diversity were reported for CGM representations; sleep physiology estimates ranged from 0.011 for retinal artery width to 0.526 for chronological age.For scalar tasks, ΔR2 compares a ridge probe using demographics plus a representation against a same-run demographic-only ridge probe on the same participants and folds; the examples differ in cohort, outcome, and representation and are not rankings.Source: §A.2 · p. 13
  • Across 18 CGM-summary evaluations covering 16 tasks, fasting-glucose ΔR2 was 0.169 and retinal artery tortuosity was −0.009; across 27 WatchPAT-summary evaluations covering 15 tasks, QT-interval ΔR2 was 0.271 and later-visit retinal artery fractal dimension was −0.014.These are evaluation rows, not independent tasks; intervals quantify within-row uncertainty and are not adjusted for multiplicity or used to classify discoveries and null results.Source: §A.2 · pp. 13–14
  • For fasting glucose, matched question-first point estimates ranged from −0.048 for ECG and −0.029 for WatchPAT to 0.035 for DXA and 0.145 for CGM.Chronological-age estimates ranged from 0.032 for CGM to 0.505 for WatchPAT; comparisons are within each question because the questions use different cohorts, and the intervals are not paired route-difference intervals.Source: §A.2 · p. 14
  • Fasting-glucose CGM estimates vary by representation, split, and cohort: 0.195 for Chronos-2 on validation participants, 0.145 on the matched four-measurement validation cohort, and test-cohort estimates of 0.169 for engineered summaries and 0.201 for Chronos-2.The validation cohorts had n=2,380 and n=1,816, respectively; the test cohort had n=6,843.Source: §A.2 · p. 14

B.1. Analysis population and model roles

  • Figure 3’s complete test-set comparison contains 160 cells across 52 tasks, with each cell sharing its target, cohort, feature set, split, and all ten methods.Twenty cells across ten tasks were excluded because at least one method was absent; these were availability exclusions, not performance exclusions.Source: §B.1 · p. 14
  • The tabular comparison includes ridge, tuned XGBoost and CatBoost, and pretrained methods including TabSwift, TabICL, TabDPT, TabPFN v2, Google TabFM, and TabPFN 3.5.XGBoost and CatBoost used fixed 12-candidate grids selected by five-fold training-set cross-validation and were evaluated once on held-out test data; the pretrained methods used pinned implementations or checkpoints.Source: §B.1 · pp. 14–15
  • The 63 exploratory engineered-feature pairs comprise 31 CGM comparisons and 32 exact-intersection WatchPAT reruns, and are not leaderboard rows.The three pretrained models with verified parameter counts were displayed in ascending count order, but the display is not a validated total capacity ordering.Source: §B.1 · p. 15

B.2. Task-level and cell-level method comparisons

Across tasks, pretrained tabular models had a positive mean advantage over ridge; same-signal Chronos-2 comparisons varied by modality and training size. The appendix also tested tool-using models’ retrieval and comparison of frozen benchmark evidence.

B.2. Task-level and cell-level method comparisons

  • Both tuned tree libraries ranked below every pretrained model under the two-test correction rule, while the top three pretrained models were not separated from one another.Neither tree library was separated from ridge in the task-level pairwise sensitivity.Source: §B.2 · p. 15
  • 0.0103 mean advantage over ridge (95% task-bootstrap interval, 0.0071–0.0136) was positive in 43 of 52 tasks.Giving each task equal weight produced this result; tasks shared participants and related outcomes, so the checks do not establish independent population replications.Source: §B.2 · p. 17

B.3. Same-signal time-series comparison

  • Panel D compared frozen zero-shot Chronos-2 representations with engineered features from the same signal, with both read by ridge; it did not compare raw signals.The ridge head was fitted to embeddings with 770–3,076 dimensions.Source: §B.3 · pp. 17–18
  • Chronos-2’s engineered-feature advantage narrowed with training size; in the larger half, its comparison was not significant for glucose but remained significant for sleep.After dropping the fatty-liver-index comparison, signed-rank p values were 0.00299 for glucose and 9.31 × 10^-9 for sleep; larger-half results were p=0.597 for glucose and p=0.000214 for sleep.Source: §B.3 · p. 18
  • A separate exploratory binary extension covered 12 matched cells across six tasks; only seven of 36 pairwise contrasts survived both Holm-adjusted tests.The type-2-diabetes DXA cell had 13 test positives, below the 17-per-class evidential threshold, and was not published on the board.Source: §B.3 · p. 18

C. Machine-querying PhenoBench evidence

  • An additional evaluation tested whether tool-using language models could retrieve and compare evidence from a frozen PhenoBench snapshot.The appendix reports this as an evaluation separate from the tabular-model and time-series comparisons.Source: C. Machine-querying PhenoBench evidence · p. 18

C.1. Results

Tool-using models varied in their accuracy across benchmark-query operations. Task Cards improved context-question accuracy for most tested models, while verdict accuracy was generally higher than full-output fidelity.

C.1. Results

  • Claude Sonnet 4.5 averaged 94.9% lookup accuracy, 93.1% method-selection accuracy, 88.9% information-source accuracy, and 91.7% context-integrated selection accuracy.Other models were uneven across operations: Opus 4.8 achieved 96.4% lookup accuracy but 51.1% method-selection accuracy, while Haiku 4.5 achieved 58.7% and 83.9%, respectively.Source: §C.1 · p. 18
  • Task Card access raised context-question accuracy for six of seven models by 22.9–68.8 percentage points, while Pixtral fell from 27.1% to 0% after a tool-loop failure.Six models returned every comparison verdict correctly, but exact verdict-plus-reason-plus-evidence fidelity ranged from 25.0% to 100%.Source: §C.1 · p. 18
  • The results show that some tested models achieved high exact fidelity while others failed selectively, but do not establish open-ended scientific reasoning, clinical decision quality, or general model superiority.Reading a result, selecting within a controlled comparison, and preserving its evidence were distinct capabilities.Source: §C.1 · pp. 18–19

C.2. Evaluation design

The evaluation used a reviewed question bank and fixed tool interface to test distinct retrieval, comparison, context-use, and comparability operations. Scoring and reporting kept those constructs separate.

C.2. Evaluation design

  • The frozen-snapshot question bank contained 277 items across five operations, including 174 method comparisons and only nine eligible information-source comparisons.The nine information-source questions exhausted eligible controlled comparisons in the snapshot and were interpreted cautiously.Source: §C.2 · p. 19
  • Seven Bedrock-hosted model versions answered every question in three epochs under fixed prompts, tools, output limits, and message limits.Temperature was zero where exposed; the Opus endpoint did not expose that setting, and Pixtral routing was prospectively changed from US to EU after quota failures.Source: §C.2 · p. 19
  • Scoring required exact row and metric values for lookup, complete candidate-row evidence for controlled comparisons, and the correct card fact and selection or abstention for context questions.Comparability items were scored primarily by supported/unsupported verdict, with reason codes and row evidence reported separately.Source: §C.2 · p. 19
  • Each model and construct was reported as the arithmetic mean and range across three epochs; repeated epochs were not treated as independent observations.No cross-construct endpoint was prespecified for inference; the five-operation equal-weight mean in the figure was added post hoc for visual compression only.Source: §C.2 · p. 19

C.3. Limitations

The machine-querying experiment measures exact adherence to a frozen benchmark interface, with results bounded by its tested models, prompts, tools, and snapshot. The figure separates accuracy across query operations and contrasts context use with full-output fidelity.

C.3. Limitations

  • The experiment measures exact adherence to a frozen benchmark interface, not open-ended scientific reasoning or clinical utility.High interface accuracy does not establish PhenoBench-LLM performance, autonomous research, or general model superiority.Source: §C.3 · p. 19
  • Results may depend on the prompts, tool schema, evaluation harness, provider implementation, and reviewed PhenoBench snapshot.Model–tool failures characterize the tested model–harness combination rather than the model alone.Source: §C.3 · p. 19, Figure 1 · p. 20
  • Coverage was narrow for information-source selection, with nine eligible cases; the Task Card analysis used eight reviewed facts encoded as matched satisfied and unsatisfied questions.Because the questions required information absent from the leaderboard, the ablation tested use of necessary card context, not whether Task Cards generally improve research quality.Source: §C.3 · p. 19
  • Figure 1C’s equal-weight mean across five operations is descriptive and post hoc, not a prespecified endpoint or general model ranking.The bars show mean exact accuracy across three epochs, with whiskers indicating the epoch range.Source: Figure 1 · p. 20

D.1. Extended results: PhenoBench-LLM

The PhenoBench-LLM extension evaluates 14 language models on shared HPP evidence across existing tasks. The accompanying methods define task-specific cohorts, matched comparisons, and reviewable model provenance.

D.1. Extended results: PhenoBench-LLM

  • 14 language models answered 40 existing Tasks across phenotype recovery, classification, follow-up forecasting, and participant ordering using a frozen prompt and shared participant evidence.Tasks used 150–480 validation participants each.Source: §D.1 · p. 21
  • The evaluation used one run per cell, 150–480 validation participants per Task, a fixed summary-feature packet, and one prompt family.The paper frames this as a controlled capability-profile, efficiency-tradeoff, and clinically relevant failure-mode evaluation.Source: §D.1 · pp. 21–22

E.1. Cohort and measurements

  • HPP combines clinical, molecular, imaging, and wearable phenotyping, while each Task specifies its eligible subset, target, timepoint, required measurements, and quality-control rules.Unless a Task specifies otherwise, participants are assigned to canonical 70/15/15 splits, repeated observations stay grouped by participant, and preprocessing is fitted on training data only.Source: §E.1 · p. 26

E.2. Benchmark tracks and model comparisons

  • The regression capacity analysis compared ten methods in 160 complete-case test cells spanning 52 tasks, matching target, participants, features, and split while varying the fitting method.The methods included ridge, tuned XGBoost and CatBoost, RealMLP, and six frozen pretrained tabular models.Source: §E.2 · p. 26
  • The time-series comparison paired frozen Chronos-2 embeddings with engineered features from the same signal, holding task, cohort, split, ridge strategy, and penalty grid fixed.No ECG pair was possible because the corpus had no engineered ECG feature set; the analysis compared representation routes under a linear head.Source: §E.2 · pp. 26–27

E.3. Statistical analysis

  • Task-level summaries averaged within each task before giving the 52 tasks equal weight, and uncertainty intervals resampled whole tasks rather than treating repeated cells as independent.The analysis also retained task-level and cell-level sensitivity summaries, with paired tests corrected separately by Holm.Source: §E.3 · p. 27

E.4. Clinically grounded evaluation tasks

  • Tasks began with HPP publications and clinician-authored surrogate targets, then underwent clinical review and were frozen into executable contracts specifying target, cohort, timing, splits, metrics, and baseline.Validity and implementation readiness were assessed separately, so valid but not yet safely executable questions could be retained for future work.Source: §E.4 · pp. 27–28

E.5. Model onboarding and model cards

  • Model Cards document contributed model families and versions, while declared training exposure informs ranking eligibility and the release protocol records predictive claim boundaries.The protocol requires a signed review, one held-out test evaluation, re-derived numbers, and a second-owner leaderboard commit.Source: §E.5 · pp. 28–29

E.6. PhenoBench-LLM: external prediction evaluation

PhenoBench-LLM evaluates language-model predictions on existing PhenoBench tasks using fixed evidence packets, response contracts, and conventional comparators. Its benchmark suite organizes these evaluations by task type and reports descriptive comparisons with explicit scope boundaries.

E.6. PhenoBench-LLM: external prediction evaluation

  • The two baseline targets are evaluated under paired information arms, with leakage exclusions, canonical participant splits, and ridge probes or other task-specific comparators.For HbA1c and visceral adipose tissue mass, the arms include demographics, CGM, NMR, and their combination; HbA1c also has a shuffled-packet control.Source: §E.6 · pp. 29–30
  • For ranking, each prompt orders four participants, and pairwise accuracy is scored over retained within-group pairs against ridge and demographic comparators.Groups are formed by seeded sampling and permutation; tied target pairs are excluded, and failed groups are also assessed with an abstain-scored sensitivity analysis.Source: §E.6 · pp. 30–31
  • Across 11 additional regression tasks, the evaluation compares LLM information-arm changes with ridge changes and interprets R^2 alongside native-unit error and rank correlation.Card-conditioned changes can reflect HPP reference values as well as clinical context; off-scale outputs can also produce large negative R^2 values.Source: §E.6 · p. 31

E.7. PhenoBench-LLM as a benchmark suite

  • PhenoBench-LLM organizes existing tasks into five categories, adding fixed evidence packets and response contracts without creating new target constructions.The categories are phenotype recovery, classification, longitudinal forecasting, ordering, and card-conditioned context.Source: §E.7 · p. 32
  • The suite spans phenotype recovery, classification, follow-up forecasting, and ordering, with task eligibility and cohorts constrained by packet compatibility and available complete evidence.The reported categories include 11 further regression tasks, five classification phenotypes, four follow-up tasks, and six group-ranking tasks.Source: §E.7 · p. 32
  • Classification uses AUROC as its primary metric, while longitudinal forecasting predicts follow-up values from baseline evidence and compares against baseline carry-forward.Because follow-up values are strongly autocorrelated with baseline values, the paired comparison with carry-forward is the informative comparison described for forecasting.Source: §E.7 · pp. 32–33
  • The suite uses revised task-card and ranking prompts, with the v2 card excluding benchmark reference values so card lift reflects curated domain context.The earlier v1 card-arm results remain as sensitivity rows rather than being re-scored.Source: §E.7 · pp. 33–34
  • The leaderboard summarizes task-level and category-level comparisons but is descriptive, does not form a composite score, and does not support general model rankings.Its single-run validation results do not establish held-out performance; language-model coverage varies across tasks, with some models run on prespecified subsets.Source: §E.6 · p. 31, §E.7 · pp. 33–34

F. Extended limitations

The paper identifies limits in task coverage, model comparisons, and the cohort-grounded language-model evaluation. It also distinguishes internal evidence from independent validation needed for claims beyond HPP.

F. Extended limitations

  • The broad but uneven question set reflects tasks formulated and implemented so far, and some comparisons estimate a combined measurement-and-representation route rather than measurement alone.Screening associations do not establish causality or clinical utility, and HPP recruitment, protocols, and missingness may limit transfer to other cohorts.Source: F. Extended limitations · p. 35
  • Model comparisons cover only compatible tasks, and the 160 cells are nested across related tasks rather than independent clinical questions.The cross-task synthesis was post hoc; shared participants and outcomes, unresampled model fitting, and representation-dimension imbalance constrain interpretation.Source: F. Extended limitations · p. 35
  • The language-model evaluation uses one run per cell, validation cohorts of 150–480, fixed summary-feature packets, and one prompt family.Its results do not establish held-out performance, general model superiority, or clinical utility.Source: F. Extended limitations · p. 35
  • Participant-level holdout prevents within-analysis participant overlap but does not ensure independence after repeated use of the same cohort.External-cohort validation remains necessary for claims intended to generalize beyond HPP.Source: F. Extended limitations · p. 35

G. LLM usage disclosure

  • Language models were also used for task curation, editing, and verification, with outputs and analyses reviewed by the authors.Source: G. LLM usage disclosure · p. 35
Loading 2609.06080v2…