Auditing Multimodal LLM Raters: Central Tendency Bias in Clinical Ordinal Scoring

Jiaqing Zhang; Sandeep Elluri; Bhanu Cherukuvada; Yonah Joffe; Jessica Sena; Miguel Contreras; Scott Siegel; Subhash Nerella; Catherine Price; Parisa Rashidi

Source-linked AI summary

Auditing Multimodal LLM Raters: Central Tendency Bias in Clinical Ordinal Scoring

Jiaqing Zhang, Sandeep Elluri, Bhanu Cherukuvada, Yonah Joffe, Jessica Sena, Miguel Contreras, Scott Siegel, Subhash Nerella, Catherine Price, Parisa Rashidi

arXiv:2605.16386v1May 11, 2026cs.CV

TL;DR

Clinical ordinal scoring by multimodal LLM raters is poorly understood, and aggregate agreement metrics may miss clinically important endpoint errors. This study audits three LLM families on Clock Drawing Test images and finds consistent compression of predictions toward the scale midpoint.

Problem
Multimodal LLM scoring behavior on clinical ordinal scales is poorly understood, while aggregate rater metrics may conceal failures at clinically important scale extremes.
Method
The study audits three multimodal LLM families against supervised deep learning models for Shulman-rated Clock Drawing Test images across two independent cohorts.
Results
All tested LLMs show central tendency bias, over-predicting low scores and under-predicting high scores despite strong tolerance-based agreement.
Takeaways & Limitations
LLM-rater evaluation should explicitly assess endpoint behavior and score compression because aggregate agreement alone can mask clinically important scoring errors.
Takeaways & Limitations
The findings remain limited to Shulman-style scoring and CDT tasks, with broader validation across alternative rubrics and clinical ordinal ratings left for future work.

Abstract

from arXiv · show

Multimodal large language models (LLMs) are increasingly explored as automated evaluators in clinical settings, yet their scoring behavior on ordinal clinical scales remains poorly understood. We benchmark three frontier LLM families against supervised deep learning models for scoring Clock Drawing Test (CDT) images on two public datasets using the Shulman rubric. While fully fine-tuned Vision Transformers achieve the best calibration (MAE 0.52, within-1 accuracy 91%), zero-shot LLMs remain competitive on tolerance-based agreement (GPT-5 MAE 0.67, within-1 accuracy 92%) despite higher absolute error. However, per-score analysis reveals that all three LLM families exhibit a pronounced central tendency effect (systematic endpoint compression): predictions are systematically compressed toward the middle of the scale, with over-prediction at the low end (score 0 to 1) and under-prediction at the high end (score 5 to 4). This effect disproportionately affects the clinically critical extremes where accurate scoring most impacts screening decisions for cognitive impairment. Targeted ablations show that neither few-shot exemplars spanning the full score range nor removing clinical terminology from the prompt eliminates the effect. Our findings extend the LLM-as-a-judge bias literature from NLP evaluation to clinical assessment, and highlight the need for calibration-aware evaluation and post-hoc calibration before deploying LLM-based raters in high-stakes screening workflows.

1 Introduction

This study audits multimodal LLMs as clinical ordinal raters using Clock Drawing Test scoring and finds consistent central-tendency bias, with predictions compressed toward the scale midpoint. The bias persists across prompting variants and disproportionately affects score extremes, exposing limitations of aggregate agreement metrics.

Audit motivation: The study proposes an audit protocol combining per-score error analysis with aggregate evaluation to reveal clinically important failures hidden by overall agreement.A rater may achieve strong aggregate agreement while systematically failing at scale extremes.
Study design: The evaluation benchmarks three commercial LLM families against established deep learning approaches on NHATS, with external validation on an independent CDT cohort.The task uses ordinal scoring of Clock Drawing Test images, which require holistic interpretation of hand-drawn imagery.
Principal finding: All tested LLMs exhibit central-tendency bias, compressing predictions toward the score-range midpoint and producing disproportionate errors at the extremes.Poor drawings are overestimated, while strong drawings are underestimated.
Bias ablations: Neither few-shot exemplars spanning the full score range nor removing clinical terminology from prompts eliminates the central-tendency effect.These ablations suggest the bias reflects intrinsic model behavior rather than a prompt-engineering artifact.
Broader significance: The findings suggest that avoiding extreme scores may generalize to structured LLM assessment settings where tail errors carry disproportionate consequences.The empirical study focuses on cognitive-impairment screening through the CDT.

2 Related Work

Prior automated CDT scoring addressed scalability and subjectivity through engineered features and newer representation-learning models, while multimodal LLMs expanded into clinical and neuropsychological assessment. Existing LLM-as-a-judge research identified central tendency bias mainly in NLP, leaving its clinical impact and prompt-level mitigation unexamined.

Automated CDT scoring: Manual CDT scoring is subjective, varies in inter-rater agreement, and does not scale to population-level screening.These limitations motivated automated CDT-scoring research.
Automated CDT scoring: Recent automated CDT-scoring work has explored Vision Transformers and self-supervised RF-VAE models for clock-drawing understanding.Earlier approaches used hand-crafted contour, digit-placement, and hand-angle features.
Clinical multimodal LLMs: Multimodal LLMs have been piloted for radiology, histological grading, broader diagnosis, CDT scoring, and speech-based cognitive screening.These models can approach expert-level performance on structured examinations but remain weak in fine-grained image interpretation and calibration.
LLM-as-a-judge bias: LLM-as-a-judge research has identified position, verbosity, self-preference, and central-tendency biases in automated evaluation.The central-tendency effect compresses score distributions toward the scale center by avoiding extreme ratings.
Research gap: Prior central-tendency findings come almost exclusively from NLP settings, leaving clinical ordinal rating, prompt mitigation, and screening consequences systematically unexamined.This study addresses the gap through controlled clinical scoring, targeted ablations, and analysis of errors at clinically consequential scale extremes.

3 Method

The study frames Clock Drawing Test automation as ordinal scoring from images and evaluates supervised CNN/ViT models alongside rubric-driven multimodal LLM raters. It uses participant-stratified NHATS data plus an independent Thai cohort, with balanced sampling for per-score analysis.

Dataset: 63,351 NHATS images from Rounds 1–13 and 386 independent Thai cohort images provide primary and external-validation datasets.NHATS comprises Medicare beneficiaries aged 65 and older; the external cohort contains paper-based clock drawings.
Dataset: 597 NHATS test images form a score-balanced benchmark by sampling 100 images per score level, except score 0 with all 97 available drawings.Development and test splits use an 80:20 participant-level stratification to prevent leakage from repeated longitudinal drawings.
Task definition: The task predicts an integer y ∈{0, 1, 2, 3, 4, 5} matching human labels, where scores are ordered and extremes carry clinical importance.Scores 0–1 may signal cognitive impairment, while score 5 indicates intact function.
Model paradigms: The comparison spans supervised CNNs, supervised token-based ViTs, and rubric-driven multimodal LLM raters using in-context reasoning without target-dataset gradient training.CNNs learn spatial features, ViTs use patch-level self-attention, and LLMs interpret images through a six-level natural-language rubric.
Supervised models: Each model outputs a discrete score ˆy or continuous estimate ˜y, with continuous predictions rounded to the integer 0–5 axis for evaluation.The CNN baseline is a pretrained ResNet-101 using cumulative ordinal modeling, while ViT variants use ordinal or bounded-regression heads.
LLM protocol: GPT-5, GPT-5.4, Gemini-2.5-Pro, and Claude-4-Sonnet receive fixed 0–5 rubrics and images, with zero-shot prompting as default and GPT-5 few-shot testing using 30 exemplars.The few-shot exemplars include 5 images per score level across the full ordinal range.

4 Experimental Setup

The study evaluates supervised deep models and multimodal LLM judges for six-level Clock Drawing Test scoring on a shared held-out benchmark of 597 images. Methods are compared under the same rubric and 0–5 output axis using calibration, tolerance-based agreement, and binary operating-characteristic metrics.

Datasets and benchmark: 597 scored images form the shared held-out benchmark for cross-family comparison.Experiments use NHATS Clock Drawing Test images with reference scores on a six-level ordinal scale from 0 to 5.
Compared methods: The comparison includes CNN, ViT-Ordinal, ViT-Continuous, GPT-5, GPT-5.4, Gemini-2.5-Pro, and Claude-4-Sonnet under the same CDT rubric.Deep models are trained or fine-tuned on NHATS images, whereas LLMs score images through direct multimodal inference from prompt inputs.
Evaluation metrics: MAE and RMSE measure absolute scoring error, within-1 accuracy measures predictions within one score level, and binary operating characteristics support comparison with prior CDT screening analyses.All methods output scores on the same 0–5 axis and are evaluated with identical downstream metrics.

5 Results

Fully fine-tuned ViT models achieve the strongest absolute calibration, while zero-shot LLMs remain competitive on within-1 agreement. Across score distributions, confusion patterns, ablations, and external replication, LLMs exhibit persistent endpoint compression toward the scale midpoint.

Aggregate performance: MAE 0.52, RMSE 0.87, and within-1 agreement (91%) make ViT-Ordinal (unfrozen) the strongest supervised model.ViT-Continuous (unfrozen) is second-best, with MAE 0.65 and within-1 89%.
Aggregate performance: MAE 0.67 and within-1 92% make GPT-5 the strongest zero-shot LLM, with within-1 agreement comparable to ViT-Ordinal (unfrozen) (91%) despite higher MAE.GPT-5.4 has MAE 0.75, and Gemini 2.5 Pro has MAE 0.84; none outperform fully fine-tuned ViTs on absolute calibration.
Score-level error patterns: LLM predictions compress toward the center: scores 0 and 5 are under-predicted, while intermediate scores, especially 1 and 4, are over-represented.Because most errors are off by one level, this compression inflates within-1 agreement while exact-match accuracy suffers.
Score-level error patterns: GPT-5’s endpoint errors concentrate in the (0→1) and (5→4) cells, while errors at true scores 2–3 are smaller and more directionally balanced.GPT-5 predicts score 5 for true score 4 in 26 cases but assigns score 5 only 22 times when the true score is 5; Gemini 2.5 Pro and Claude 4 Sonnet show the same pattern more severely.
Targeted ablations: MAE drops from 0.67 to 0.56 and within-1 agreement rises from 92% to 94% with few-shot exemplars, but endpoint compression persists.Accy=5 improves from 22% →52%, whereas Accy=0 improves only from 35.0% →41.2%; removing clinical framing instead worsens MAE from 0.67 →0.82 and within-1 from 92% →87%.
External replication: The Thai CDT cohort reproduces the same toward-center error structure, with low true scores over-predicted and high true scores under-predicted.Endpoint errors are again asymmetric, supporting the persistence of extreme-rating under-assignment outside the in-domain setting.

6 Discussion

Aggregate agreement can conceal systematic endpoint compression in multimodal LLM clinical ordinal scoring, with low scores over-predicted and high scores under-predicted. This clinically asymmetric failure limits standalone use when reliable endpoint identification matters and motivates broader validation and calibration-aware adaptation.

Core finding: Aggregate agreement metrics can obscure endpoint compression, with low scores over-predicted and high scores under-predicted toward the scale center.GPT-5’s within-1 agreement can be comparable to supervised models despite this systematic deviation.
Clinical implications: Extreme-score errors are clinically asymmetric: under-scoring severe impairment may reduce case detection, while shifting intact drawings from 5 to 4 may increase false alarms and review burden.The discussion therefore cautions against standalone ordinal CDT rating when endpoint identification is important.
Broader significance: The findings extend LLM-as-a-judge bias research from NLP response evaluation to clinical absolute scoring by identifying endpoint compression under rubric-based ordinal assessment.The pattern is presented as a related failure mode in a patient-relevant ordinal task.
Limitations and future work: Validation remains limited to Shulman-style scoring on two CDT cohorts, off-the-shelf LLMs, and a limited set of prompt variants.Future work should examine alternative rubrics, non-CDT clinical ordinal tasks, post-hoc calibration, and task-specific adaptation.

7 Conclusion

The study evaluated multimodal LLMs as raters for six-level Clock Drawing Test scoring across two independent cohorts, comparing them with supervised deep learning models. Although LLMs achieved strong tolerance-based agreement, their predictions consistently compressed toward the scale’s midpoint, over-predicting low scores and under-predicting high scores.

Evaluation scope: The evaluation covered multimodal LLM raters, supervised deep learning models, a six-level clinical ordinal scale, and two independent Clock Drawing Test cohorts.The comparison focused on automated CDT scoring across distinct clinical datasets.
Main finding: Despite strong tolerance-based agreement, frontier LLM predictions showed a consistent central tendency effect.Per-score analysis exposed systematic compression that aggregate agreement alone may not reveal.
Scoring bias: LLMs over-predicted low scores and under-predicted high scores, compressing predictions toward the middle of the scale.The endpoint errors occurred in opposite directions at the lower and higher ends of the ordinal scale.

A Technical Appendices and Supplementary Material · A.1 Supervised Model Details

The supervised models use a shared, reproducible training protocol combining staged optimization, standardized image preprocessing, and deterministic execution settings.

A.1 Supervised Model Details: Two-phase optimization first aligns the head with a frozen backbone, then performs full fine-tuning at a reduced learning rate.This staged protocol is shared across all supervised models.
A.1 Supervised Model Details: 224×224 resizing and ImageNet-statistics normalization standardize the input images.
A.1 Supervised Model Details: Fixed random seeds and deterministic CUDA settings support reproducibility.

A.1.1 CNN (ResNet-101 Ordinal)

The CNN uses a pretrained ResNet-101 with a five-logit cumulative ordinal head for Clock Drawing Test scoring. Training combines staged fine-tuning with imbalance correction, while optional preprocessing extracts the clock to reduce background clutter.

Preprocessing: Optional clock extraction reduces background clutter through grayscale conversion, Otsu thresholding, morphological cleanup, component selection, and padded cropping.Component selection uses density and aspect ratio before cropping.
Training: Training proceeds from a frozen-backbone head phase to full-network fine-tuning with separate backbone and head learning rates.Phase 1 uses SGD at 10^-3 for 15 epochs; Phase 2 uses backbone learning rate 10^-5 and head learning rate 10^-3 for 100 epochs.
Training: Class imbalance is addressed with inverse-frequency weights and a weighted random sampler.The training procedure also uses momentum and weight decay.
Architecture and ordinal output: A pretrained ResNet-101 feeds a Linear(2048, 5) head that produces five cumulative ordinal logits.The logits correspond to score thresholds for ordinal prediction.

A.1.2 ViT-Ordinal … A.5 Full Confusion Matrices

The appendix specifies supervised ViT ordinal and continuous training, deterministic multimodal LLM inference and evaluation procedures, full prompt variants, and confusion-matrix analyses. Across these analyses, LLM predictions show toward-center endpoint compression that persists under few-shot and de-clinicalized prompting.

A.1.2 ViT-Ordinal: ViT-Ordinal uses a pretrained ViT backbone with a cumulative ordinal head and two-phase AdamW training, selecting checkpoints by validation quadratic-weighted Cohen ’s κ.Training uses learning rates 10−3 for head-only training and 5×10−6 for full-model training, with 10 and 20 epochs respectively.
A.1.3 ViT-Continuous: ViT-Continuous replaces the ordinal head with a scalar regression head, clamps outputs to [0, 5], and selects checkpoints by validation MAE.Its optimizer and training schedule are identical to ViT-Ordinal.
A.2 Multimodal LLM Technical Details: LLM requests include 0–5 score definitions, JSON-only instructions, base64-encoded target images, and deterministic decoding with temperature = 0 and top-p=1.Few-shot mode prepends five labeled reference images per score level while preserving the single-JSON-score output contract.
A.2 Multimodal LLM Technical Details: The LLM pipeline uses bounded parallelism, rate throttling, up to three exponential-backoff retries, JSON parsing with regex fallback, and score clamping to [0, 5].Malformed responses are marked invalid if score recovery fails.
A.3 Reproducibility Notes: All paradigms are evaluated on the same score-balanced 597-image benchmark using MAE, RMSE, exact-score accuracy, within-1 agreement, specificity, and sensitivity.Confidence intervals use 2,000 bootstrap iterations over prediction–label pairs; supervised training and evaluation use a single NVIDIA B200 GPU, while LLM inference uses commercial APIs.
A.4 Full LLM Prompts: The clinical prompt defines NHATS-style scores 0–5, prohibits negative or administrative codes and reasoning text, and requires valid JSON containing only the score.The de-clinicalized ablation retains the same score definitions but frames the evaluator as an image quality assessment expert.
A.5 Full Confusion Matrices: The complete 6×6 confusion matrices cover supervised models, zero-shot clinical LLM judges, and prompt ablations on the 597-image benchmark.The matrices are organized as Figures 4–6.
A.5 Full Confusion Matrices: LLM confusion matrices concentrate errors in the 0→1 and 5→4 cells, with the pattern persisting across prompt conditions despite few-shot gains for GPT-5 at score 5.Gemini 2.5 Pro predicts score 5 only 3 times out of 100 true-5 drawings; GPT-5 few-shot performance increases from 22 →52, while de-clinicalized score-5 accuracy drops near zero.