PaperlayerView exact revision on arXiv

Source-linked AI summary

Documenting Trofinetide Treatment Phenotypes in Rett Syndrome with a Dual-LLM Consensus Pipeline

Daniel Palacios, Guantong Qi, Mikael Guzman Karlsson, Shawntel K. David, Mohammed Sameeruddin, Bernhard Suter, Zhandong Liu, Davut Pehlivan, Hu Chen

arXiv:2610.22062v1q-bio.QMcs.CL

TL;DR

Real-world monitoring of trofinetide response across Rett syndrome features is limited by labor-intensive chart review and incomplete trial coverage. This study applies a dual-LLM consensus pipeline to clinical notes, with a severe-diarrhea prompt achieving 84.2% accuracy against expert labels and outperforming regular-expression matching. The approach extends to 17 additional phenotypes, although those results are supported by spot-check review rather than full ground-truth validation.

Abstract

from arXiv · show

Objective: Trofinetide, approved in 2023 as the first Rett syndrome treatment, lacks scalable real-world monitoring because chart review is labor-intensive. We developed a dual large language model (LLM) consensus pipeline to extract 18 treatment-response phenotypes, comprising 4 adverse effects, 13 Rett clinical features, and medication refusal, from clinical notes. Materials and Methods: We analyzed 26,763 notes from 108 patients with confirmed sustained trofinetide exposure at Texas Children's Hospital. The pipeline filters notes by keyword, classifies each through two LLMs requiring consensus, and aggregates a patient-by-timepoint matrix. Architecture and prompt were selected on a diarrhea development set, then frozen and applied to the other 17 phenotypes with the phenotype definition substituted, as an out-of-sample test. The reported prompt comparison scores 183 expert-labeled cells. Results: A prompt requiring diarrhea to be active and severe performed best (84.2 $\pm$ 0.0% accuracy; F1 = 0.674), 47 percentage points above regular-expression matching ($p < 0.0001$). Expert reviewers judged six of the pipeline's calls per phenotype, three positive and three negative, and agreed with 90.2% (92/102). The 13 Rett clinical features are manifestations of the disorder itself rather than adverse effects of trofinetide. Gross and fine motor impairment, language/communication, and stereotypy were documented present in 92-100% of patients at every visit. Discussion: In-context LLM extraction captured multi-domain treatment responses. The outcome is documented presence at each visit, not a direction-of-change judgment. Conclusion: Coupling in-context LLM reasoning with a consensus architecture offers a scalable alternative to keyword and regular-expression extraction for real-world evidence, adapting to new phenotypes and medications through configuration alone.

INTRODUCTION

Rett syndrome causes progressive loss of acquired abilities and affects multiple functional domains. Although trofinetide became the first FDA-approved treatment in 2023, trial coverage and duration leave real-world responses across Rett features incompletely characterized.

INTRODUCTION

  • Rett syndrome is a severe neurodevelopmental disorder primarily caused by MECP2 loss-of-function mutations and affecting approximately 1 in 10,000–15,000 live female births.It typically involves apparently normal early development, regression with loss of hand skills and spoken language, stereotypic hand movements and gait abnormalities, followed by stabilization.Source: INTRODUCTION · p. 3
  • Trofinetide became the first FDA-approved Rett syndrome treatment in March 2023, after Phase 3 improvements on the RSBQ and CGI-I scales.The trial captured only a subset of treatment-relevant phenotypes over 12 weeks, and real-world response across Rett features remains incompletely characterized.Source: INTRODUCTION · p. 3

BACKGROUND AND SIGNIFICANCE

Monitoring treatment response in Rett syndrome requires tracking multiple domains documented across narrative notes, where rule-based approaches can struggle to distinguish current symptoms from historical, negated, or anticipatory mentions. This study uses independently prompted LLMs with a consensus requirement to extract multi-domain phenotypes at scale.

BACKGROUND AND SIGNIFICANCE

  • Manual review of narrative notes spanning motor, cognitive, behavioral, and autonomic domains is labor-intensive, while traditional rule-based NLP can confuse active symptoms with historical, anticipatory, or negated mentions.Earlier EHR phenotyping approaches include concept-extraction systems and rule-based or portable phenotype algorithms.Source: BACKGROUND AND SIGNIFICANCE · p. 3
  • A single LLM produced false positives and false negatives in this data, motivating independent model verification for judging whether findings are active at a visit.The two independently prompted models must agree on a positive call, prioritizing precision over recall.Source: BACKGROUND AND SIGNIFICANCE · p. 3
  • The pipeline extracts 18 phenotypes from notes for 108 patients, developing and evaluating on expert-annotated diarrhea before applying the frozen approach to 17 other phenotypes with only the definition substituted.The phenotypes comprise four adverse effects, 13 Rett clinical features, and medication refusal.Source: BACKGROUND AND SIGNIFICANCE · p. 3

MATERIALS AND METHODS

The retrospective study analyzed a QC-confirmed trofinetide cohort using a five-stage, dual-LLM pipeline that classifies documented phenotype presence by patient and timepoint. Diarrhea served as the development and quantitative evaluation phenotype; the frozen configuration was then applied to the other phenotypes.

Study design and population

  • Quality control confirmed sustained trofinetide exposure in 108 of 115 identified patients by triangulating pharmacy orders, administration records, and longitudinal note mentions.Seven patients without confirmed exposure were excluded; 65 confirmations came from pharmacy orders and 43 from clinical documentation alone.Source: Study design and population · p. 4
  • The cohort contributed 26,763 analyzed notes, with a median of 148 notes per patient across the full cohort.The diarrhea-validation subset had more notes per patient than the remaining cohort (median 171 vs. 99, p = 0.07); cohort median age was 13 years and participants were almost entirely female.Source: Study design and population · p. 4
  • The study integrated registry, clinical-note, and two structured medication-table sources; relying only on structured orders would have missed patients whose specialty-pharmacy dispensing bypassed that table.Forty-three of 108 confirmed patients had no matching Epic order-table row.Source: Data sources · p. 4

Extraction and evaluation

  • The five-stage pipeline anchors follow-up to treatment start, buckets notes into six-month windows, filters by phenotype keywords, classifies notes with two LLMs, aggregates calls, and evaluates diarrhea against expert review.Nine patients lacked a reliable exposure date and were excluded from timepoint analysis; 183 diarrhea cells from 56 patients were evaluable, including 47 labeled positive.Source: Pipeline architecture · pp. 5–6
  • Each keyword-positive note receives independent calls from a clinical-NLP model and a domain specialist, and counts as positive only when both say YES.The classifier distinguishes current from historical, negated, anticipatory, and templated mentions; for diarrhea, the definition also imposes a severity threshold.Source: Pipeline architecture · p. 5, Phenotype definitions · p. 6

Annotation procedure

  • Expert annotation used a consensus-adjudication workflow without formal written guidelines or a retrospective inter-annotator agreement statistic, limiting gold-standard reliability.The LLMs ran with platform-default decoding parameters, which were not guaranteed deterministic; the study reports variability across three runs.Source: Pipeline architecture · p. 6

Phenotypes

  • A phenotype is marked present at a visit when it is documented in that visit’s note, rather than based on a judgment of change over time.The 18 selected phenotypes comprise four trofinetide adverse effects, medication refusal as a separate behavioral response, and 13 Rett clinical features that are not treatment adverse effects.Source: Phenotype definitions · p. 6

Model development and iteration

  • The diarrhea architecture and prompt were selected on the same 183 cells used for scoring, making its reported accuracy an in-sample upper bound rather than a held-out estimate.The remaining 17 phenotypes provide the out-of-sample transfer test after the pipeline was frozen and only the phenotype definition substituted.Source: Model development and iteration · p. 7

RESULTS

The severe-diarrhea prompt had the strongest headline performance against the expert-labeled ground truth, substantially exceeding rule-based baselines, though its advantage over other LLM prompts was not statistically resolved. The pipeline was also applied across 17 additional phenotypes, whose documentation patterns and expert spot-check agreement provide broader but less fully validated results.

Contextual reasoning improves performance and outperforms keyword matching

  • 84.2% accuracy, precision 0.714, F1 0.674, and κ 0.570 were achieved by the severe-diarrhea prompt, 47 percentage points above regular expressions at 37.2% accuracy.Its accuracy was also 42 points above negation-aware NLP at 42.1%; Table 2 compares seven methods on the same 183 expert-labeled cells.Source: Contextual reasoning improves… · p. 7, Table 2 · p. 8
  • All prompt variants significantly outperformed both rule-based methods (p < 0.0001), but none of the prompt-to-prompt comparisons remained significant after Holm correction.The severe prompt won 9 cells and lost 1 against the deployed prompt, with p = 0.021 before and p = 0.15 after correction; the cohort was too small to resolve the roughly eight-cell gap.Source: Contextual reasoning improves… · p. 7
  • A patient-level bootstrap placed accuracy’s 95% CI at 78.1–89.9%, and accuracy was 79.1% on the 129 on-treatment cells with the ranking unchanged.The 54 baseline cells were almost all true negatives and raised every configuration’s accuracy equally; baseline accuracy was 96.3%.Source: Contextual reasoning improves… · p. 8
  • Predicted diarrhea prevalence was lower than expert labels at six months (45.5% vs. 63.6%) and higher at 24 months (30.8% vs. 15.4%).The authors attribute discrepancies mainly to annotation and data-availability factors, including stricter human thresholds for managed diarrhea and evidence falling in an adjacent window.Source: Contextual reasoning improves… · p. 8

The pipeline extends to 18 phenotypes over 30 months of follow-up

  • Expert reviewers agreed with 90.2% of pipeline calls across the 17 non-diarrhea phenotypes (92/102), with agreement of 86.3% on positive and 94.1% on negative calls.Per phenotype, nine reached 6/6 agreement, six reached 5/6, and dysautonomia and medication refusal reached 4/6.Source: The pipeline extends to 18 phenotypes… · p. 9, Table 3 · p. 9

Adverse-effect and clinical-feature presence follow distinct temporal patterns

  • Other-side-effect documentation rose from 11% at baseline to 58% at six months and settled at 38–49% thereafter, while weight-loss documentation remained 62–71% despite being 63% before treatment.Medication refusal peaked at 38% at 18 months and later fell to 15–16%; vomiting peaked at 38% at 12 months and fell to 22% by 30 months.Source: Adverse-effect and clinical-feature… · p. 9
  • Gross- and fine-motor impairment, language/communication impairment, and stereotypy were documented in 92–100% of patients at every visit.These 13 clinical features are manifestations of Rett syndrome, not adverse effects; baseline-to-follow-up documentation differences should not be read as clinical change, and features other than diarrhea were not validated against ground truth.Source: Adverse-effect and clinical-feature… · pp. 9–10

Most residual discrepancies reflect differing scoring targets rather than misreading of the note

  • Among diarrhea discrepancies, the largest category was definitional disagreement about clinically significant diarrhea (14/29, 48%), and the 29 errors comprised 17 false negatives and 12 false positives.The rule-based taxonomy separates evidence by window but cannot distinguish within-note misreading from window-placement mismatch; re-scoring threshold disagreements yields a 91.8% bounding sensitivity-analysis estimate, not a corrected performance figure.Source: Most residual discrepancies reflect… · p. 10, Table 4 · p. 10

DISCUSSION

The discussion presents dual-LLM consensus as a scalable way to extract treatment-response phenotypes, while emphasizing that performance and transfer evidence have important limits. The analysis captures documented presence, not whether features improved or worsened.

DISCUSSION

  • The severe-diarrhea prompt was strongest, but its margin over the deployed prompt is unresolved and its accuracy is an in-sample upper bound.It was selected and scored on the same 183 cells.Source: DISCUSSION · p. 11
  • 90.2% agreement on the other 17 phenotypes and a half-cohort split support transfer of the frozen architecture and prompt design.The phenotype definition alone was substituted.Source: DISCUSSION · p. 11
  • Most diarrhea errors reflect annotation or data limitations rather than model failures, distinguishing prompting fixes from data-coverage ceilings and annotation-guideline decisions.The pipeline detects only what the corpus documents; symptoms may appear in other channels.Source: DISCUSSION · p. 11
  • Only diarrhea has a ground truth; the other 17 phenotypes rely on spot checks and inter-LLM agreement, so their rates are hypothesis-generating lower bounds.Generalization may vary by phenotype, with concrete adverse effects expected to transfer more readily than subjective findings.Source: DISCUSSION · p. 11
  • The pipeline captures documented presence rather than direction of change, and observed trajectories cannot be definitively attributed to trofinetide.Concurrent therapies and natural development may also contribute; core-feature improvement would require severity-graded instruments such as the RSBQ or CGI-I.Source: DISCUSSION · p. 11

CONCLUSION

The study applies a dual-LLM consensus pipeline to trofinetide treatment-response extraction in Rett syndrome. It outperforms a regular-expression baseline on the diarrhea development set and transfers to 17 additional phenotypes with the phenotype definition substituted.

CONCLUSION

  • The pipeline was applied to 108 patients with Rett syndrome receiving trofinetide over 30 months.It extracts multiple treatment-response phenotypes from clinical notes.Source: CONCLUSION · p. 12
  • The pipeline substantially outperforms a regular-expression baseline on an expert-annotated diarrhea development set.Source: CONCLUSION · p. 12
  • The same configuration transfers to 17 other phenotypes with only the phenotype definition substituted.The conclusion describes extension to further phenotypes, medications, and institutions by configuration alone.Source: CONCLUSION · p. 12

ADDITIONAL CONTRIBUTIONS

The authors report that AI tools were not used for study design, analysis, or primary manuscript drafting. Generative AI was used only to proofread and correct author-written text.

ADDITIONAL CONTRIBUTIONS

DATA AVAILABILITY

The pipeline, phenotype keyword lists, and LLM prompts are publicly available, but the underlying clinical notes cannot be shared. De-identified aggregate results are provided in the article and supplementary materials.

DATA AVAILABILITY

  • The classification pipeline, phenotype keyword lists, and LLM prompt templates are publicly available.Source: DATA AVAILABILITY · p. 12
  • Underlying clinical notes cannot be shared because of patient-privacy protections and institutional/HIPAA policy.Source: DATA AVAILABILITY · pp. 12–13
  • Access to de-identified derived data may be requested from the corresponding author, subject to a data-use agreement and IRB approval.Source: DATA AVAILABILITY · p. 13
  • De-identified aggregate results supporting the figures and tables are provided in the article and supplementary materials.Source: DATA AVAILABILITY · p. 13

FIGURES

The figures trace the extraction pipeline and cohort, compare diarrhea-validation methods, and show documented presence and expert spot-check agreement across other phenotypes.

Figure 1

  • Figure 1 depicts five stages from keyword filtering through dual-LLM consensus and patient-by-timepoint aggregation, with evaluation against ground truth.The cohort panel shows 115 patients screened, 108 with QC-confirmed exposure, and 56 in the expert-annotated diarrhea ground truth; the phenotype panel groups 18 phenotypes.Source: Figure 1 · p. 17, FIGURES · p. 17

Figure 2

  • 84.2% accuracy was highest among seven methods, versus 37.2% for regular-expression matching; the two severity-bar prompts scored 83.1% and 84.2%.Figure 2A compares accuracy and F1 across methods; the reported configuration was identical across three runs.Source: Figure 2 · p. 18, FIGURES · p. 19
  • Figure 2’s 29 errors were attributed to threshold disagreement (48%), temporal displacement (41%), and unavailable note evidence (10%).The figure also shows per-timepoint confusion and expert-versus-predicted prevalence.Source: Figure 2 · p. 18, FIGURES · p. 19

Figure 3

  • Fine and gross motor, stereotypy, and language/communication were documented in nearly all patients throughout follow-up.The heatmap spans baseline through 30 months; Rett clinical features are disorder manifestations, not drug-related harm.Source: FIGURES · p. 19, Figure 3 · p. 19
  • Weight-loss documentation was high at baseline and remained high, while medication refusal and other side effects were rare at baseline and rose afterward.Source: FIGURES · p. 20
  • Among 17 phenotypes, expert spot-check agreement was 6/6 for nine, 5/6 for six, and 4/6 for two.Each phenotype had six checked cases: three model-positive and three model-negative.Source: Figure 3 · p. 19, FIGURES · p. 20

SUPPLEMENTARY NOTES

The supplementary material documents the pipeline’s agreement, consensus, and prompt-development methods and qualifies interpretation of its diarrhea results. It addresses benchmark and annotation reliability, exposure-date assignment, error-taxonomy limits, intended use, and reporting-checklist scope.

Supplementary Note S1. Inter-LLM Agreement and the Consensus Gate

  • κ = 0.930 with 96.6% raw agreement across 60,530 shared note–phenotype instances; diarrhea separately reached κ = 0.878 and 98.0% raw agreement across 2,260 notes.Agreement between independently prompted agents is a proxy for how well specified a phenotype is, not a measure of expert agreement; spot-check results did not consistently track κ.Source: Supplementary Note S1. Inter-LLM… · p. 21
  • 38.4% of instances were positive only when both agents said YES, while the gate treated one-sided calls as negative to suppress false positives at some cost in recall.Against expert labels, both-NO cells were 84.3% accurate (113/134), split votes 72.7% (8/11), and both-YES cells 52.3% (23/44); positive calls were proposed for human adjudication.Source: Supplementary Note S1. Inter-LLM… · pp. 21–22

Supplementary Note S2. Model Development History

  • 79.6% accuracy for the consensus gate versus 79.2% for the stronger single agent on the refreshed 183-cell grid differed by two cells.Architecture and prompt accuracies were selected and reported on the same grids, so they are in-sample estimates; the consensus gate was adopted for its observed performance and visibility into disagreement, not proven necessity.Source: Supplementary Note S2. Model… · pp. 22–24

Supplementary Note S3. LLM Prompt Templates

  • A V2 prompt rule requiring diarrhea to be active and clinically significant was worth 4.6 percentage points against expert annotation, but the difference did not survive multiplicity correction.V2 specifies that mild, controlled, or adequately managed diarrhea is NO and requires ongoing concern or a management change for YES.Source: Supplementary Note S3. LLM Prompt… · p. 26

Supplementary Note S4. Reading the Error Taxonomy, and Intended Use

  • The error taxonomy cannot identify within-window note misreadings, and the 91.8% figure is a bound rather than corrected accuracy.The reported diarrhea operating point—precision 0.714 and recall 0.638—is intended for cohort-level prevalence and trajectory estimation and reviewer triage, not autonomous per-patient decisions.Source: Supplementary Note S4. Reading the… · pp. 26–27

Supplementary Note S5. TRIPOD-LLM Reporting Checklist (Addressed Items)

  • The study reports against TRIPOD-LLM but lists addressed items rather than providing a complete attestation of all 19 items and 50 subitems.Items specific to de novo model development, instruction tuning, or model updating are not applicable because the study uses an existing externally developed LLM through prompting.Source: Supplementary Note S5. TRIPOD-LLM… · p. 27
  • The TRIPOD-LLM checklist identifies lack of an inter-annotator agreement statistic as a gold-standard reliability limitation because cells were resolved jointly by consensus.Source: Supplementary Note S5. TRIPOD-LLM… · p. 30

Supplementary Note S6. Annotation Procedure

  • The clinical team labeled diarrhea and severity by consensus, without independent annotation or a written guideline, preventing retrospective inter-annotator agreement statistics.In the partial spot-check overlap with evaluation windows, expert agreement was 91.4% (53/58); the full note sample’s reported agreement was 90.2% (92/102).Source: Supplementary Note S6. Annotation… · p. 30

Supplementary Note S7. Prompt-Level and Variable-Decomposition Sensitivity Analyses

  • Accuracy rose from 79.6% with deployed prompt V0 to 84.2% with V2, but the difference was not significant after Holm correction (p = 0.15).False positives fell from 21 to 12 while true positives remained essentially unchanged at 31 and 30; the difference was 9 cells won against 1 lost.Source: Supplementary Note S7. Prompt-Level… · p. 31
  • Explicit reasoning rules did not improve V2 (83.1% versus 84.2%), and decomposing extraction into two questions yielded 61.7%–79.4%, below the reported configuration’s 84.2%.The deployed prompt already encoded the resolution and alternative-cause rules; in the decomposition, the onset variable’s 61.7% accuracy and 0.384 precision constrained the combinations.Source: Supplementary Note S7. Prompt-Level… · pp. 31–32
  • Counting medication-controlled diarrhea as present worsened performance by 4.4 points for V1 and 4.6 points for V4, adding 14 false positives to remove 6 false negatives.Neither broader version was significantly worse than V0 after correction, so the deployed threshold is supported by consistent direction across runs rather than a resolved test.Source: Supplementary Note S7. Prompt-Level… · pp. 31–32
  • Onset-aligning predictions raised modal-run accuracy from 79.8% to 84.2% with F1 = 0.707, but this bound assumes diarrhea begins at the first on-treatment visit.A visit-number-only benchmark scored 82.5%, and 35 of 47 positive cells occurred at 6 months, indicating substantial timing structure in the labels.Source: Supplementary Note S7. Prompt-Level… · p. 33

Supplementary Note S8. The Exposure Anchor

  • Replacing the order date with earliest documented exposure shifted the median anchor by 116 days and reduced the evaluation grid from 189 to 183 cells.The corrected anchors preserved the 47 positive cells; the deployed prompt’s accuracy rose from 77.1% to 79.6%, but the simultaneous corpus refresh prevents attributing that rise to anchor correction alone.Source: Supplementary Note S8. The Exposure… · pp. 33–34

SUPPLEMENTARY TABLES

The supplementary tables describe the note and phenotype datasets, model agreement, prompt sensitivity, and the scope of the reported visit-level findings.

SUPPLEMENTARY TABLES

  • 26,763 of the 91,453 retrieved notes fell inside evaluation windows and formed the analyzed subset for 108 patients.The diarrhea funnel further distinguishes keyword-passing notes from notes classified by both agents.Source: Table S1 · p. 35
  • The keyword lists shown are abbreviated; the full lists contain 14–22 terms per phenotype and 24 terms for diarrhea.Source: Table S2 · p. 35
  • Stereotypy was documented in 97% of classified notes, while 99.4% raw agreement still yielded a capped κ because prevalence was near-universal.The passage reports that this prevalence caps κ; it does not provide a κ value here.Source: SUPPLEMENTARY TABLES · p. 36
  • All three runs produced identical predictions on the 183 cells for the reported diarrhea configuration, with 30 true positives, 124 true negatives, 12 false positives, and 17 false negatives.The underlying note-level calls were not fully stable.Source: Table S5 · p. 37, Table S6 · p. 37
  • Adverse-effect table entries are percentages documented present at each visit, with variable denominators; diarrhea is scored separately using its validated severe-diarrhea prompt.Cells based on one patient are suppressed.Source: Table S7 · p. 37
  • Rett clinical-feature entries describe documented presence, not improvement or worsening; baseline-to-6-month differences should not be read as clinical change.Baseline documentation was lower than at 6 months for five of the 13 features.Source: Table S8 · p. 38
  • The agreement table covers 60,530 valid note–phenotype matches for the 17 phenotypes using the presence prompt; diarrhea is evaluated separately with its own prompts.The diarrhea comparison uses the same 2,260-note corpus for both prompts.Source: Table S9 · p. 38
  • The diarrhea sensitivity analysis found the severity threshold was the only prompt element that materially moved accuracy; no decomposition rule beat the deployed prompt.No prompt-family contrast cleared Holm correction, so V2 is described as the top of a sweep rather than a demonstrated improvement.Source: Table S4 · p. 36, Table S10 · p. 39

B. Onset/control decomposition, ten combination rules

This subsection describes variants that decompose onset and control variables and alter their combination or within-window aggregation.

B. Onset/control decomposition, ten combination rules

C. Temporal alignment of the V0 predictions

The temporal-alignment variants shift model predictions or both predictions and annotations, while the supplementary comparison also reports architecture-selection context and inter-model agreement.

C. Temporal alignment of the V0 predictions

  • S1 shifts model output so its first predicted positive aligns with the first on-treatment visit, then pads the tail as negative.Source: C. Temporal alignment of the V0… · p. 39
  • S3 aligns predictions and annotations to their own first positive visit, making it positional rather than deployable.Source: C. Temporal alignment of the V0… · p. 39
  • The S1 contrast is against V2 rather than V0.Source: C. Temporal alignment of the V0… · p. 39
  • The earlier architecture comparison selected the two-agent consensus configuration, but the authors describe it as best-performing rather than proven necessary.A final rerun placed the gate within 0.4 points of the stronger single agent, differing on two of 183 cells.Source: Table S11 · p. 40
  • Across 60,530 matches, the consensus gate counted only both-YES outcomes as positive and classified one-sided disagreements as negative.The figure reports 38.4% both-YES outcomes and 3.4% one-sided disagreements.Source: Figure S1 · p. 41
Loading 2610.22062v1…