A bilingual AI audiologist built through rubric-guided closed-loop engineering outperforms human audiologists in a blinded evaluation of simulated cases

Introduction

• LLMs’ strong knowledge-benchmark and audiogram results did not establish end-to-end consultation ability; reviewed audiology systems lacked such evaluation, and bilingual assessment was rare.Audiology consultation requires structured history-taking, test interpretation, diagnosis and management, and patient-centred communication.Source: Introduction · pp. 1–2

A standardised bilingual audiology consultation environment

• The bilingual environment used 73 training cases and a matched AI patient simulator; the pre-specified 60-case evaluation subset yielded 58 primary-analysis cases after two exclusions.The primary set included 30 Chinese and 28 English cases, and AI and human audiologists interacted with the same simulator instance.Source: A standardised bilingual audiology… · p. 3, Figure 1 · p. 4

Rubric-guided closed-loop engineering builds specialist consultation behaviour

• Rubric-guided closed-loop engineering converted consultation feedback into a frozen 19-rule playbook without updating language-model parameters.The loop scored consultations, extracted reusable lessons, curated playbook rules, and re-evaluated the system on training cases.Source: Rubric-guided closed-loop engineering… · pp. 4–5, Figure 2 · p. 5

The AI audiologist outperforms human audiologists in blinded evaluation

• 58/58 cases favored the AI (mean paired Δ = +1.35, Cohen’s d = 1.84); it also outperformed humans on 20 of 21 rubric items and in both languages.The primary comparison used simulated consultations in Chinese and English.Source: Introduction · p. 3, The AI audiologist outperforms human… · pp. 6, 8, Figure 4 · p. 7

Rating agreement supports the robustness of the primary comparison

• The AI–human advantage remained positive for every rater and every leave-one-rater-out analysis, despite moderate single-rater agreement.Leave-one-rater-out overall composite differences ranged from +1.10 to +1.60; two-rater ICC(2, k = 2) for the overall composite was 0.73.Source: Rating agreement supports the… · pp. 9–10, Figure 5 · p. 9

Rubric-guided playbook augmentation is the largest contributor to the gain

• Playbook augmentation was the largest contributor: it improved GPT-5 history-taking and communication, while tools and retrieval added little or no rubric-level gain.The history-taking gain replicated on Qwen3.5-Plus (ΔF1 = +0.66 vs +0.65 for GPT-5); retrieval alone had no overall gain, and adding tools and retrieval to the playbook produced ΔF1 = −0.05 and ΔF2 = −0.04. These ablations were exploratory and used automated rubric evaluation.Source: Rubric-guided playbook augmentation is… · pp. 10–11, Figure 6 · p. 10

Discussion

• The paper presents explicit, experience-derived behavioural guidance as a route to specialist consultation agents in low-data domains where workflow knowledge is rich but case material is sparse.The full system retained image interpretation and grounded retrieval for audiogram images and authoritative patient-facing materials, although these components were not the main source of rubric-level gains.Source: Discussion · pp. 11–12
• Several induced rules reflect simulator constraints or require clinical revision before deployment, supporting explicit policy audit and clinician-led oversight.Examples include the 30-turn cap, psychiatric diagnosis from screening scores, and suppressing re-escalation after specialist review; the frozen rules were reported as tested, not as clinically validated.Source: Discussion · pp. 12–14
• Interpretation is limited by simulated consultations, a finite case set, moderate rater agreement, rubric-related circularity, and language comparisons confounded by rater and comparator assignment.The primary analysis used 58 cases; the between-language gap narrowed when the Chinese-only rater was excluded. Source materials cannot be fully released, and cross-backbone analysis covered one alternative model using automated evaluation; future work includes prospective real-patient evaluation and broader domains.Source: Discussion · pp. 13–14

Case sources and standardisation

• Cases came from eight licensed teaching and case sources, were standardised into a bilingual schema, and were partitioned into separate training and evaluation sets.The primary blinded comparison used 58 cases after two incomplete English transcripts were excluded; no identifiable patient data were processed, and case content was de-identified or fictionalised teaching material.Source: Case sources and standardisation · p. 14, Ethics statement · p. 14

AI patient simulator

• The AI patient simulator progressively disclosed case facts in lay language and used the same configuration for AI and human consultations.It had no access to ground-truth diagnoses or management, and test results were released only after a named-test tool request, with no releases on the first turn.Source: AI patient simulator · p. 15

AI audiologist system architecture

• The GPT-5-based system combined an externally maintained playbook, multimodal audiology interpretation, retrieval-augmented grounding, and orchestration without updating language-model weights.Audiogram and other test interpretation used structured vision parsers, while otoscopic images used a ResNet50 classifier; retrieval indexed four English audiology textbooks.Source: AI audiologist system architecture · p. 15

Rubric-guided closed-loop engineering

• Rubric-guided closed-loop engineering used two consultations per training case and a curator-managed playbook lifecycle, producing a frozen 19-rule GPT-5 policy for evaluation.The evaluator–reflector scored transcripts against a 21-item rubric and generated reusable guidance; the same frozen playbook was used in the blinded comparison and GPT-5 component ablation.Source: Rubric-guided closed-loop engineering · p. 16

Human audiologist consultations

• Seventeen practising audiologists produced consultations under matched simulator conditions, while blinded raters assessed anonymised AI and human transcripts using a prespecified 21-item rubric.Each of 58 cases had one consultation per source, each rated independently by two raters; the overall composite was the unweighted mean of history-taking, patient-centred communication, and diagnosis-and-management Likert dimensions.Source: Human audiologist consultations · p. 17, Human rating protocol · pp. 17–18

Automated evaluation

• Blinded human ratings—not automated rubric evaluation—formed the primary AI–human comparison, with paired tests specified for overall, dimension, language, and item-level analyses.The automated evaluator supported training feedback and component ablation only; the primary endpoint was the case-level paired overall-composite difference, with leave-one-rater-out robustness analyses specified.Source: Automated evaluation · pp. 18–19, Statistical analysis · p. 19

Inference settings and compute

• GPT-5 reasoning was set to low during 73-case engineering and medium for evaluation consultations, with each consultation capped at 30 turns and 30 audiologist-agent requests.Patient-simulator requests were capped at eight per turn; conversation runs used provider-default sampling temperature and maximum output tokens. Runs occurred between March and early May 2026, using hosted commercial APIs except for the locally run otoscopy classifier.Source: Inference settings and compute · p. 20

Reporting

• The study followed TRIPOD-LLM, supplemented by CONSORT-AI and DECIDE-AI.Source: Reporting · p. 20

Data availability

• Derived analysis tables are available from the corresponding author on reasonable request, while raw case materials, transcripts, and identifiable operational records are not publicly released.Additional derived materials may be provided to journal editors or qualified academic researchers subject to licensing, confidentiality, consent, and institutional-approval constraints.Source: Data availability · p. 20

Code availability

• The research-prototype system’s code is not currently open source, but is planned for release after further clinical validation.The system is documented in the Methods and Supplementary Information, including the evaluator–reflector specification and frozen 19-rule GPT-5 policy; machine-readable seed files and other analysis materials are available to editors and reviewers under confidential review.Source: Code availability · p. 20

Competing interests

• LL, CM, CL and HY are affiliated with Orka Labs Inc.; all other authors declare no financial or non-financial competing interests.LL, CM and CL are employees, and HY is an intern, at the hearing-technology company; these authors declare no non-financial competing interests.Source: Competing interests · p. 23

Patient and public involvement

• Patients and the public were not involved, and evaluation cases came from de-identified or fictionalised teaching materials rather than real-patient encounters.Source: Patient and public involvement · p. 23

Study registration

• The study was not publicly pre-registered, but its analysis plan was internally pre-specified before raters viewed transcripts or analyses were conducted.The plan covered the primary endpoint, language-stratified analyses, rater assignment, exclusions, and bootstrap and multiple-comparison procedures.Source: Study registration · p. 23

Supplementary Information

• The supplement includes eight notes covering case sources, the patient simulator, the rubric, engineering, ablations, human ratings, and sensitivity and case-level analyses.Source: Supplementary Information · p. 23

Source materials

• Cases came from eight published compendia and training texts spanning five audiology content domains, with source titles withheld under licensing restrictions.Audiologists standardised cases while blinded to later partitioning, consulting cited clinical guidelines when source materials lacked ground-truth diagnoses or management plans; incomplete fields were recorded rather than imputed.Source: Source materials · p. 25, Cross-source consistency · pp. 26–27

Structured case schema

• Each source-agnostic case schema contains clinical content, diagnostic and management ground truth, and metadata, enabling matched presentation without source-identifying tokens.The schema includes presenting complaint, history, examination and audiometric findings, as well as diagnosis, management, investigations, red flags, follow-up, language, case type, and complexity markers.Source: Structured case schema · pp. 25–26

Overview

• The separate GPT-5-mini patient simulator uses the same case-specific instance for AI and human consultations, limiting its knowledge to the structured patient card and answering in the case language.It cannot access the ground-truth diagnosis, management plan, rubric, audiology reference corpus, playbook, or audiologist image-interpretation tools.Source: Overview · p. 28, Model and access · p. 28, State-bounded knowledge · p. 28

Behavioural constraints

• The simulator gives short, lay-language responses, avoids inventing facts or interpreting results, and releases a test only when the audiologist requests it by name.Runtime controls block first-turn test release, cap releases at one per turn, and limit the patient agent to eight requests per turn; implementation details are available to editors and reviewers on request.Source: Behavioural constraints · p. 29, Tool-gated test release · p. 29, Reproducibility · p. 32

21-item rubric

• The rubric covers F1 history-taking and clinical reasoning (5 items), F2 patient-centred communication (6 items), and F3 diagnosis and management (3 Likert/ordinal and 7 binary safety items).F1 is grounded in audiology best-practice recommendations [ref 37 in main bibliography]; F2 adapts the Kalamazoo Consensus Statement [ref 6] and is informed by the Calgary–Cambridge guide [ref 38].Source: 21-item rubric · p. 34
• The overall composite is the unweighted mean of F1, F2 and F3-Likert, while F3-YN counts Yes-marked safety items on a 0–7 scale.F3-Likert averages M1, M2 multiplied by 1.25, and M3; M2's possible values are 1.25, 2.50, 3.75 and 5.00.Source: 21-item rubric · p. 34

Item anchors

• Supplementary Table S2 specifies item anchors and exemplars, including 1–5 anchors for Likert items, four ordinal labels for M2, and Yes criteria for M4–M10.The rubric also serves as the feedback structure for the automated evaluator–reflector and the structure of human rating.Source: Item anchors · p. 34, 21-item rubric · p. 34, Table S2 · p. 36

Combined evaluator–reflector feedback module

• The combined evaluator–reflector scored consultations, generated item-level rationales, tagged helpful or harmful rules, and extracted reusable lessons for curator updates.It also provided a supporting metric for component-ablation analyses, but was not the endpoint for the primary AI–human comparison, which relied exclusively on blinded human rating.Source: Combined evaluator–reflector feedback… · pp. 34–35
• The automated module enabled scalable training-time adaptation and supported mechanistic component analyses.The module's scoring component was used in the component-ablation analyses.Source: Combined evaluator–reflector feedback… · pp. 34–35

Rubric circularity considerations

• Δ = +0.90 to +1.92 across Likert and ordinal items; the shared rubric role and frozen rules matching specific rubric behaviours raise a circularity concern.Blinded human rating does not by itself resolve the concern; the non-significant M8 difference (Δ = +0.03, q = 0.47) should not be interpreted as evidence against rubric-score maximisation.Source: Rubric circularity considerations · p. 35

• M1 scores the appropriateness of the leading differential diagnosis from 1 (unsafe / irrelevant) to 5 (correct, prioritised, evidence-aligned).Source: F3 Likert — Diagnosis & management… · p. 40
• M3 scores the overall management plan from 1 (unsafe / irrelevant) to 5 (correct, prioritised, evidence-aligned).Source: F3 Likert — Diagnosis & management… · p. 40
• M2 uses four anchored categories for differential-diagnosis comprehensiveness, with scores multiplied by 1.25 for composite computation, yielding 1.25, 2.50, 3.75 and 5.00.Source: F3 Likert — Diagnosis & management… · p. 40

• Seven safety items are scored Yes when the criterion is satisfied and No otherwise, using the same criteria for human raters and the automated evaluator.Source: F3 Binary — Safety items (Yes/No) · p. 41
• F1 averages H1–H5, F2 averages C1–C6, and F3-Likert averages M1, M2 weighted by 1.25, and M3.The weighted M2 values are 1.25, 2.50, 3.75, or 5.00.Source: F3 Binary — Safety items (Yes/No) · p. 42
• F3-YN counts Yes responses among M4–M10 on a 0–7 scale.Source: F3 Binary — Safety items (Yes/No) · p. 42
• The primary endpoint is the unweighted mean of F1, F2, and F3-Likert on a 1–5 scale.Source: F3 Binary — Safety items (Yes/No) · p. 42

Overview

• Across 73 training cases, a state-machine loop runs two consultations per case, with a curator update between attempts.The loop pairs the AI audiologist and its current playbook with an AI patient simulator, combined evaluator–reflector feedback, and a curator.Source: Overview · p. 43

Per-case workflow

• The first attempt is scored on 21 rubric axes to identify helpful or harmful rules and generate reusable candidates; the second tests the updated playbook.The second attempt uses the same patient simulator and case, compares axis-level outcomes with the first, and may refine rules but adds no backlog candidates.Source: Per-case workflow · p. 43

Playbook schema

• Each playbook item stores a trigger–action–effect rule, its section and ID, usage and helpfulness metadata, and its backlog, staging or core layer.Sections are tactics, strategy or common_mistakes; metadata includes usage and promotion iterations and staging TTL remaining.Source: Playbook schema · pp. 43–44

Three-layer memory architecture

• The three-layer memory keeps up to 30 unsurfaced candidates in backlog, while staging items expire after four iterations and core holds up to 20 proven rules.Core and staging items are surfaced for prompting; backlog items are not. When full, core replaces its lowest-scored item only if the promoted item clears a score margin.Source: Three-layer memory architecture · p. 44

Curator 6-step pipeline

• At each iteration, the Curator updates rule-use metadata, ingests candidates into backlog, and uses an LLM review to select up to three actionable, novel, high-impact staging items.Backlog IDs are sequential, and the oldest candidates are pruned above capacity 30; selected items receive a four-iteration TTL.Source: Curator 6-step pipeline · p. 44
• Staging rules lose one TTL iteration per cycle and expire at zero; they qualify for core when helpfulness minus harmfulness reaches 2.If core is full, a candidate must exceed the lowest-scored core rule by 0.1; core overflow evicts the lowest-scored items to a discarded-items log.Source: Curator 6-step pipeline · p. 45

Core scoring formula

• Core-rule retention weights behavioral helpfulness most heavily, alongside recency and normalized usage.CORE_SCORE = 0.35 × recency + 0.20 × usage_term + 0.45 × help_term; helpfulness is floored at 0, and recency decays by a factor of 0.9 per iteration.Source: Core scoring formula · p. 45

Combined evaluator–reflector (turn 1) — verbatim system prompt

• The evaluator receives clinical expectations and an AI-generated conversation, then scores each of the 21 rubric axes using concrete excerpts or described behavior.It is instructed to justify each score with evidence from the conversation.Source: Combined evaluator–reflector (turn 1)… · pp. 45, 47
• Its structured output provides per-axis evaluations, actionable rules, question templates, clinical rationales, and an overall coaching summary.The required JSON schema also requests error identification, root-cause analysis, a correct approach, and a key insight.Source: Combined evaluator–reflector (turn 1)… · pp. 46–47
• The evaluator ranks the 3–5 axes needing most improvement and tags generator playbook bullets as helpful or harmful.The ranking is based on where the conversation most needs improvement.Source: Combined evaluator–reflector (turn 1)… · p. 47
• Backlog candidates must be concise, actionable trigger-to-effect proposals addressing observed, generalizable failures without duplicating existing rules or contradicting the audiologist prompt.Candidates are assigned to strategy, tactics, or common_mistakes.Source: Combined evaluator–reflector (turn 1)… · pp. 46–47
• The combined evaluator–reflector uses Gemini 3.1 Pro Preview through Google Vertex AI under the same hosted-API regime as the curator.Source: Combined evaluator–reflector (turn 1)… · p. 47

• The evaluator receives ground truth, the turn-1 conversation and its per-axis scores, and the turn-2 conversation for the same case.Source: Combined evaluator–reflector (turn 2)… · p. 47
• Playbook edits must not contradict the audiologist’s hard-coded rules; conflicting items must be tagged harmful.Source: Combined evaluator–reflector (turn 2)… · p. 48
• It scores every rubric axis for turn 2 and justifies each score with excerpts.Source: Combined evaluator–reflector (turn 2)… · p. 49
• For each axis, it compares turn-2 scores with turn-1 scores, marking improvement, degradation, or no change and explaining the direction.Source: Combined evaluator–reflector (turn 2)… · p. 49
• It tags case-specific playbook items as helpful when they contributed to improvement or harmful when they correlated with regression, and may revise only turn-1-created items.Source: Combined evaluator–reflector (turn 2)… · p. 49

Curator backlog-review prompt — verbatim

• The curator reviews candidate playbook items and selects up to three for promotion into the active staging layer.Fewer may be selected if fewer candidates are worthy.Source: Curator backlog-review prompt — verbatim · p. 49
• Candidates are ranked by impact, novelty, and clarity: recurring failure modes, non-duplication of staging or core items, and an actionable trigger, action, and expected effect.Source: Curator backlog-review prompt — verbatim · p. 50
• The active playbook consists of staging and core items, providing the reference for judging candidate novelty.Source: Curator backlog-review prompt — verbatim · p. 50
• For each selection, the prompt requests its backlog ID and brief reasoning; if none qualify, it directs the curator to return an empty list.Source: Curator backlog-review prompt — verbatim · p. 50

Training outcome

• 19 rules comprised the frozen policy: 5 strategy, 6 tactics and 8 common_mistakes, all in core.The snapshot had no staging or backlog entries.Source: Training outcome · p. 50
• Induction used 73 training cases: 43 English and 30 Chinese; the policy was frozen before every reported playbook-equipped GPT-5 evaluation, with no inference-time updates.Complete rule text and stable IDs are in Supplementary Table S3; curator counters are induction-time bookkeeping, not study outcomes.Source: Training outcome · p. 50

Reproducibility

• The frozen evaluation playbook contains 19 rules, documented by stable ID and complete Trigger, Action and Expected-effect text in Supplementary Table S3.All 19 entries belong to the frozen core layer; the snapshot has no staging or backlog entries, and displayed IDs are non-contiguous.Source: Reproducibility · pp. 50, 53, Table S3 · p. 51
• The full evaluator–reflector, curator, playbook schema and orchestration implementation is held by the corresponding author and available to editors and peer reviewers on request under confidential review.The main manuscript’s Code availability section provides this access condition.Source: Reproducibility · p. 50

Supplementary Note 4.

• The taxonomy distinguishes short behavioural micro-rules (tactics), broader workflow priors (strategy) and failure modes to avoid (common_mistakes).Source: Supplementary Note 4 · p. 53

GPT-5 component ablation

• The GPT-5 component ablation is the note’s main analysis and underlies the rubric-level claims reported in the main text.The note identifies this analysis as Section 5.1 and links it to the main-text discussion in Results, Fig. 6.Source: Supplementary Note 5. Component… · p. 54

Cross-backbone analysis

• The exploratory Qwen-family analysis examined whether the GPT-5 component pattern also emerged on a backbone with lower baseline performance.It applied the same rubric-guided playbook procedure to test whether the pattern was specific to GPT-5.Source: Supplementary Note 5. Component… · p. 54

Evaluation setup

• Both analyses used the automated 21-item rubric on the same 75-case curated evaluation pool: 40 Chinese and 35 English cases.Source: Supplementary Note 5. Component… · p. 54

5.1 GPT-5 component ablation

• The ablation compared Vanilla, Vanilla + Tools, Vanilla + RAG, Vanilla + Playbook, and the full integrated system, using the same frozen 19-rule playbook loaded before inference.Vanilla was the minimal GPT-5 baseline; the augmentations added multimodal interpretation tools, retrieval-augmented grounding, or the playbook, individually or together.Source: §5.1 · p. 54

Evaluation protocol

• Each configuration completed one consultation per case; automated rubric scoring used a 75-case pool, except Vanilla + RAG had n = 74 after one Chinese case parser failure.Five GPT-5 records lacked one dimension composite and were not imputed; results are descriptive, with no additional retrospective inferential comparisons.Source: Evaluation protocol · p. 54, (Supplementary Note 2) · p. 54

Dimension-level results

• The ablation’s F3-L diagnosis-and-management score averaged M1 and M3, with M2 reported separately; F3-B was the proportion of M4–M10 safety items marked Yes.This F3-L differs from the blinded human evaluation’s F3-Likert composite, which also includes M2 multiplied by 1.25.Source: Dimension-level results · p. 55

Δ vs Vanilla (n-weighted)

• +0.65 on F1 and +0.94 on F2 versus Vanilla came from Vanilla + Playbook, exceeding Tools’ +0.20 on each; the full system added no gains over Playbook on F1 or F2 (−0.05 and −0.04).RAG alone yielded ΔF1 −0.10 and ΔF2 +0.02 versus Vanilla; differences between the full system and playbook were within cross-language scatter.Source: Δ vs Vanilla (n-weighted) · p. 55

Item-level results (Likert items only)

• H1 rose from 2.90 to 3.97 (+1.07) and H5 from 2.40 to 4.20 (+1.80) with Playbook, while near-ceiling Vanilla scores on H3 and H4 gained little.Playbook also raised C2 by +1.28, C5 by +1.21, and C3 by +1.09; Tools produced smaller selective gains on C3 and C4, while RAG effects across F2 were near-zero or negative.Source: Item-level results (Likert items only) · p. 56
• The component ordering on F1 and F2 was Playbook > Tools > RAG, consistent with the playbook’s behavioural scaffolding contributing more than declarative knowledge to rubric gains.The playbook encoded when to probe, how to organize the consultation, and failure modes to avoid, with gains concentrated on consultation structure and patient-centred fluency.Source: Item-level results (Likert items only) · p. 56

Setup and Qwen-specific differences from the GPT-5 ablation

• The Qwen analysis used an independently induced 7-rule playbook and disabled reasoning mode; it tested the induction procedure, not transfer of GPT-5’s 19-rule playbook.The five-configuration ablation used 75 cases in every configuration and an automated evaluator; the patient simulator, retrieval pipeline, rubric, and other listed inference settings were held constant.Source: Setup and Qwen-specific differences… · p. 57, §5.2 · p. 57

Dimension-level results (Qwen)

• Vanilla + Playbook produced Qwen’s largest single-component gains on F1 (+0.66) and F2 (+0.57), replicating the component ordering observed with GPT-5.Qwen’s Vanilla baseline was lower than GPT-5 on F1 (2.70 vs 3.54), F3-L (3.10 vs 3.69), and F3-B (0.67 vs 0.85), but similar on F2 (2.09 vs 2.12).Source: Dimension-level results (Qwen) · p. 58

Item-level results (Qwen, Likert items only)

• On Qwen, Vanilla + Playbook raised H1 from 1.71 to 2.85 (+1.15) and H5 from 1.73 to 2.71 (+0.97), while H3 and H4 were near ceiling under Vanilla.The largest playbook-driven F2 communication gains were C5 responding to emotions (+1.61), C1 fostering the relationship (+0.96), and C2 gathering information (+0.55).Source: Item-level results (Qwen, Likert items… · pp. 58–59
• On Qwen, Full added +0.84 over Vanilla + Playbook on C6, +0.33 on M1, and +0.43 on M3.These items assess written planning, differential-diagnosis appropriateness, and management-plan appropriateness, respectively; the authors suggest tools and retrieval may supply clinical content not fully encoded by the playbook.Source: Item-level results (Qwen, Likert items… · p. 59

Cross-backbone pattern

• Qwen Full exceeded Vanilla + Playbook on F2 (+0.27), F3-L (+0.38), and F3-B (+0.06), while GPT-5 configurations were essentially indistinguishable across dimensions.The Qwen Full-system advantage was concentrated on concrete clinical-content items, including written planning, differentials, and management plans.Source: Dimension-level results (Qwen) · p. 58, Cross-backbone pattern · p. 59

Limitations of the cross-backbone analysis

• The cross-backbone findings are descriptive exploratory evidence from one alternative backbone, not blinded human-rated evidence or evidence of generalisation to other backbones.No retrospective paired significance tests were introduced, and the analysis does not establish how the pattern would extend to smaller open-weight or other frontier models.Source: §5.2 · p. 57, Limitations of the cross-backbone… · pp. 59–60

Supplementary Note 6. Detailed human rating results

• The five largest Chinese–English gaps in paired AI–Human differences were H5, M3, H2, H3 and C1.Supplementary Fig. S5 plots per-item paired differences with 95% confidence intervals, ordered by the Chinese–English gap.Source: Supplementary Note 6. Detailed human… · p. 64, Figure S5 · p. 66

Inter-rater agreement (ICC and pairwise correlations)

• Single-rater ICC(2,1) values were 0.53–0.57 across the four primary scores, while two-rater-mean reliability was ICC(2,k = 2) = 0.69–0.73.Raters ranked consultations more consistently than they matched absolute score levels; the primary analysis averaged two ratings per consultation. Ordinal-item ICCs ranged 0.396–0.583 (median 0.45), versus 0.024–0.332 (median 0.19) for binary safety items, several of which had near-ceiling Yes rates.Source: Inter-rater agreement (ICC and… · pp. 67–68

Leave-one-rater-out sensitivity

• Excluding any one rater preserved the AI advantage, with paired Δ from +1.10 to +1.60, Cohen's d from 1.39 to 1.93, and all paired P values below 10^14.Excluding R3 gave the smallest estimate (+1.10); excluding R2 gave the largest (+1.60).Source: Leave-one-rater-out sensitivity · p. 69

Rater-level paired sign test

• Across 115 rater–case scoring events, AI exceeded Human in 107 (93.0%); a one-sided sign test on 112 non-tied events yielded P < 10^-20.There were five Human-better events and three ties.Source: Rater-level paired sign test · p. 70

Descriptive findings

• Thresholded low-score events comprised one AI event each on H2, H3, M4 and M10, versus Human counts of 28 on C4, 27 on C6 and 22 on C5.A low-score event meant a two-rater case mean ≤ 2.0 for Likert/ordinal items; binary failure required both raters to mark No. Thresholded counts do not identify clinical causes or transcript-level errors.Source: Operational definitions · p. 72, Descriptive findings · p. 73, Interpretive constraints · p. 73

Supplementary Note 6. Detailed human rating results

• The AI exceeded the Human on the overall composite in all 58 paired cases; M8 was the only rubric item without a significant AI advantage (Δ = +0.03, q = 0.47).The non-significant M8 difference should not be interpreted as equivalence.Source: Supplementary Note 6. Detailed human… · p. 63, Descriptive findings · p. 73

Descriptive findings

• Human-better item events were most frequent for M8 and M10, with six cases each, although AI exceeded Human on the overall composite in all 58 paired cases.Ties were common on near-ceiling binary items; item-level events are not case-level composite wins or transcript-level diagnostic errors.Source: Operational definitions · p. 72, Descriptive findings · p. 73