PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM4CKD: Large Language Models for Early Stage Chronic Kidney Disease Screening

Muhammad Ashad Kabir, Sirajam Munira

arXiv:2609.04013v2cs.AIcs.LG

TL;DR

Early CKD screening is difficult when diagnostic resources, stable feature sets, and labeled data are limited. LLM4CKD evaluates zero-shot and few-shot LLM screening with clinically selected structured prompts against ML, DL, tabular foundation-model, and screening-tool baselines. LLMs achieve competitive performance with few examples, whereas supervised and tabular models improve more consistently as training data grow.

  • Problem

    CKD screening is constrained by limited diagnostic resources, heterogeneous feature availability, and conventional methods' reliance on labeled data and stable feature spaces.

  • Method

    LLM4CKD evaluates five LLMs using clinically selected tabular features and structured prompts under zero-shot and few-shot settings, comparing them with ML, DL, tabular foundation-model, screening-tool, and independent-dataset baselines.

  • Results

    LLMs achieve competitive balanced accuracy with very few examples, while supervised ML and tabular models generally improve more predictably as labeled training data increase.

  • Takeaways & Limitations

    LLMs may complement CKD screening when labeled data are limited, whereas conventional and tabular models remain advantageous when sufficient training data and infrastructure are available.

  • Takeaways & Limitations

    Performance estimates are limited by the relatively small primary cohort, and LLM reliability varies with prompts, feature serialization, model versions, and deployment settings.

Abstract

from arXiv · show

Early screening of chronic kidney disease (CKD) is critical for timely intervention, yet most machine learning (ML) and deep learning (DL) approaches require labeled data and model training, limiting their use in real-world screening settings. This study evaluates the effectiveness of large language models (LLMs) for CKD screening under zero-shot and few-shot in-context learning settings and compares them with traditional ML and DL methods. We propose a framework that uses clinically selected tabular features and structured prompt templates to enable LLM-based inference without task-specific training. LLM performance is evaluated across multiple prompt styles, feature configurations, and data settings, and compared with standard ML, DL, and tabular foundation model (TFM) baselines, and existing CKD screening tools. The results show that LLMs can achieve competitive performance using only a small number of examples, often matching or outperforming traditional approaches in low-data settings. However, their performance remains model-dependent and less stable as input complexity increases. In contrast, ML, DL, and TFM models show more consistent improvement with larger training data. Overall, the findings highlight a trade-off between data efficiency and stability, suggesting that LLMs may serve as a flexible complementary approach for CKD screening when labeled data are limited. To facilitate further research and reproducibility, the code has been made publicly available at https://github.com/akabircs/LLM4CKD

I. INTRODUCTION

CKD is often silent and difficult to screen at scale where diagnostic resources and stable feature sets are unavailable. LLM4CKD evaluates LLMs as a low-resource, data-efficient alternative alongside established predictive approaches and screening tools.

  • CKD affects approximately 850 million people worldwide, with nearly 80% of cases residing in low- and middle-income countries.
  • Early-stage CKD can be clinically silent, while timely detection can support interventions that slow disease progression.
  • Conventional ML and DL methods rely on fixed feature spaces, curated datasets, and stable deployment conditions that real-world screening often lacks.
  • LLMs offer zero-shot and few-shot in-context learning, reducing reliance on large labeled datasets for heterogeneous screening settings.
  • LLM4CKD benchmarks LLMs against conventional ML, DL, and tabular foundation-model baselines in low-resource early-stage CKD screening.
  • The study additionally examines clinical plausibility through feature-ranking comparisons, established CKD tools, and independent-dataset evaluation.

A. Related Work and Research Gaps

Prior CKD screening research has not systematically established how LLMs perform beyond zero-shot inference, relative to diverse baselines, or across datasets and screening tools. LLM4CKD addresses these gaps through comparative and cross-dataset evaluation.

  • Existing CKD tools often emphasize later-stage disease and high-income cohorts, limiting direct applicability to LMIC early-detection settings.
  • Recent nephrology LLM studies largely focus on clinical support, knowledge extraction, communication, and hybrid feature interpretation rather than quantitative CKD screening.
  • Few-shot LLM-based CKD screening remains underexplored beyond zero-shot feature-guided inference.
  • No prior study had systematically compared zero- and few-shot LLMs with conventional ML, DL, and tabular foundation-model CKD baselines.
  • The literature also lacked systematic analysis of LLM feature importance against ML rankings, epidemiological risk factors, datasets, and established risk tools.

II. METHODOLOGY

The LLM4CKD methodology combines structured feature processing, harmonized datasets, selected-feature analysis, and comparative evaluation. It uses a Bangladeshi primary cohort and an independent Indian UCI cohort for cross-dataset assessment.

  • The framework serializes structured features, constructs prompts, performs LLM inference, analyzes clinically grounded feature rankings, and evaluates performance against tools and an independent dataset.
  • Dataset Preparation: The primary evaluation uses a community-based Bangladeshi cohort, while the independent evaluation uses the hospital-based Indian UCI CKD dataset.
  • Dataset Preparation: Dataset-2 contains 400 records, including 250 CKD cases and 150 non-CKD controls, and excludes serum creatinine to reduce label leakage.
  • Dataset Preparation: Feature harmonization standardizes names and value encodings so clinically equivalent variables support consistent prompts and comparable evaluation across datasets.
  • Dataset Preparation: The selected-feature subset comes from prior ML analysis and is mapped to Dataset-2 using variables with equivalent clinical interpretations.

B. Feature Serialization and Prompt Style

LLM4CKD converts harmonized tabular patient records into text while preserving feature semantics, then compares list-based and text-based serialization templates. These templates provide alternative structured representations for prompt-based inference.

  • Feature serialization maps harmonized clinical variables into text while preserving feature names, value encodings, and clinical semantics.
  • The same serialization procedure is applied to all-feature and selected-feature settings for comparable prompt and dataset evaluation.
  • The list-based template represents each clinical variable as an explicit key–value pair, preserving feature boundaries within a patient profile.
  • The text-based template expresses the same patient information as a concise clinical description.

C. Prompt Construction and Templates

The framework serializes harmonized clinical features into structured prompts and evaluates instruction- and chat-style inference with list- and text-based templates under zero- and few-shot settings.

  • C. Prompt Construction and Templates: Instruction-style prompts explicitly define the binary CKD classification task and expected output, whereas chat-style prompts present the same task conversationally.
  • C. Prompt Construction and Templates: Zero-shot prompts contain only the task instruction and query patient record, whereas few-shot prompts prepend labeled examples using the same serialization format.
  • D. Large Language Models Used for Inference: The evaluated LLMs include Gemma-2-9B, Llama-3-8B, Qwen-3-8B, Mistral-7B, and GPT-4o-mini.
  • D. Large Language Models Used for Inference: For few-shot inference, labeled patient-record examples are inserted before the query record in the instruction-style list template.
  • D. Large Language Models Used for Inference: The chat-style text template provides the conversational prompt format used for Llama inference.
  • D. Large Language Models Used for Inference: The model set combines compact open-weight models with the proprietary API-based GPT-4o-mini baseline across common feature settings and zero- and few-shot protocols.
  • D. Large Language Models Used for Inference: Open-weight models use both instruction- and chat-style prompts, while GPT-4o-mini is evaluated only through chat-style prompting.

E. Feature Ranking and Explainability

The study analyzes clinically plausible LLM feature prioritization alongside model evaluation procedures that use controlled prompting, probabilistic outputs, and repeated comparisons.

  • E. Feature Ranking and Explainability: A surrogate ML model and SHAP estimates are used to compare LLM risk-factor prioritization with data-driven feature importance.
  • E. Feature Ranking and Explainability: Zero-shot models are evaluated without labeled examples, training, or parameter updates, directly on the full dataset.
  • E. Feature Ranking and Explainability: Fig. 2 evaluates all- versus selected-feature inputs across prompt styles and templates using balanced accuracy and paired Brier-loss permutation tests.
  • E. Feature Ranking and Explainability: Open-weight models use deterministic decoding, while probabilistic classification converts output-token log-probabilities into normalized CKD and non-CKD likelihoods.
  • E. Feature Ranking and Explainability: TabPFN and SAINT implementations use fixed model-internal random states where applicable, with source code publicly available for the experimental pipeline.

III. RESULTS AND DISCUSSION

Results show that selected features can improve zero-shot LLM screening, while few-shot LLMs are data-efficient but less consistently scalable than supervised tabular models.

  • A. Zero-Shot Analysis of Feature Selection and Prompt Design: Selected-feature inputs improve or maintain zero-shot performance for most LLMs, with Llama-3 gaining from 0.51 to 0.74 in the Chat+Text setting.
  • B. Low-Data Comparison with ML, DL, and Tabular Foundation Models: Across selected-feature comparisons, LLMs use varying shot counts while ML and DL/TFM baselines train on corresponding labeled subsets.
  • A. Zero-Shot Analysis of Feature Selection and Prompt Design: Gemma-2 generally loses balanced accuracy with selected features, whereas prompt style and serialization effects vary across models.
  • B. Low-Data Comparison with ML, DL, and Tabular Foundation Models: Gemma-2, Qwen-3, and GPT-4o-mini reach approximately 0.80–0.81 balanced accuracy at four-shot inference, while Llama-3 and Mistral weaken or decline.
  • B. Low-Data Comparison with ML, DL, and Tabular Foundation Models: GPT-4o-mini remains around 0.80 balanced accuracy across shots, making it the most stable LLM in the comparison.
  • B. Low-Data Comparison with ML, DL, and Tabular Foundation Models: Supervised ML and tabular models generally improve as labeled training data increase, with TabPFN and NODE strongest at larger training sizes.

C. Pairwise Brier Loss Comparison

Pairwise Brier-loss comparisons identify Qwen-3 as the strongest overall LLM, while TabPFN and NODE lead tabular models and gain advantage as labeled data increase.

  • LLM comparisons: Qwen-3 shows the strongest overall probabilistic performance among LLMs, with lower Brier loss than Llama-3 and Mistral.It remains competitive with Gemma-2 and GPT-4o-mini.
  • LLM comparisons: 0.146 Brier loss is Qwen-3’s value at 32 shots, following 0.154 at four shots and 0.141 at 16 shots.
  • Tabular model comparisons: 0.820 balanced accuracy and 0.122 Brier loss make TabPFN the best overall top-model result at 32 context samples.NODE remains strong with balanced accuracy of 0.797 and Brier loss of 0.142.
  • Data-size effects: TabPFN and NODE benefit more consistently from additional labeled data than LLMs, particularly for probabilistic calibration.Selected-feature LLM prompting remains strong in low-resource settings.
  • Conventional ML comparisons: MLP remains a competitive supervised baseline, but its Brier loss ranges from 0.160 to 0.175 across training sizes.Its probabilistic calibration is weaker than the strongest LLM and tabular foundation-model baselines.

D. Feature Ranking and Clinical Plausibility

LLM-derived feature rankings are clinically plausible because they emphasize recognized CKD risk factors, although their weighting differs from the MLP reference and epidemiological patterns.

  • Reference ranking: Hypertension, age ≥60, and RBC are the top three features in the MLP reference ranking and have strong epidemiological CKD associations.
  • Ranking agreement: The feature-importance heatmap compares Spearman rank correlations between LLMs and the MLP model.Darker cells indicate stronger agreement.
  • LLM rankings: LLMs prioritize clinically meaningful CKD risk factors but weight them differently from the MLP-based reference.
  • Comparative plausibility: LLMs emphasize diabetes, anemia, and obesity more strongly, whereas the MLP reference emphasizes hypertension, age, and urinary RBC.

E. Comparison with CKD Screening Tools

Zero-shot LLMs achieve competitive performance against established CKD screening tools, while independent-dataset results show that feature selection effects vary by model.

  • Screening-tool comparison: 0.7635–0.8103 balanced accuracy is achieved by established screening tools, with Thakkinstian highest among rule-based tools.
  • Screening-tool comparison: −0.111 to −0.206 Brier-loss differences favor Qwen-3 over all five tools significantly (p < 0.001).Negative values denote lower probabilistic loss for the LLM.
  • Screening-tool comparison: 0.7925 balanced accuracy is Mistral’s highest LLM result, alongside significant Brier-loss reductions across all tool comparisons.
  • Independent-dataset evaluation: Feature selection improves or maintains performance for most models on Dataset-2, but its effect remains model-dependent.
  • Independent-dataset evaluation: 0.73 to 0.88 balanced accuracy is Llama-3’s gain on Dataset-2 with selected features (p < 0.01).Mistral and GPT-4o-mini improve slightly without statistically significant differences, while Qwen-3 decreases from 0.82 to 0.81 (p < 0.01).

G. Limitations

The study’s conclusions are constrained by cohort size, model and prompt dependence, feature-selection assumptions, limited cross-dataset evidence, and untested clinical deployment factors.

  • Study limitations: The relatively small primary cohort may limit performance-estimate stability, despite repeated runs and independent-dataset evaluation.Larger and more diverse cohorts are needed for stronger generalization claims.
  • Study limitations: LLM reliability depends on prompt design, feature serialization, model version, and token-level log-probability availability.Results may vary across future model releases and deployment settings.
  • Study limitations: The selected feature subset was derived from prior ML analysis and may not be optimal across populations or clinical contexts.
  • Study limitations: Independent-dataset results provide preliminary cross-dataset evidence rather than definitive clinical validation.The datasets differ in sampling setting, feature availability, and case-control composition.
  • Study limitations: Robustness to patient-level missing features, heterogeneous inputs, deployment efficiency, clinical safety, cost, and prospective validation was not directly evaluated.These factors require evaluation before clinical use.
Loading 2609.04013v2…