Can JEV Support Clinical Trial Criteria Assessment?

1 Introduction

• Prescreening must distinguish unmet conditions from insufficient evidence, since interpreting clinical statements depends on context, negation, and timing.A symptom may leave a criterion unresolved when its required duration is undocumented.Source: §1 · p. 1
• Prior generative approaches include TrialGPT’s criterion-level predictions with explanations and sentence locations, and smaller LLMs trained through synthetic-data distillation.These studies motivate evaluating both judgment quality and resource use.Source: §1 · p. 1
• JEV accepts patient context, natural-language questions, and predefined options, returning selected labels with native option probabilities.Source: §1 · p. 1
• The relevant comparison is whether JEV matches schema-constrained models in classification, insufficient-evidence handling, latency, and cost.Criteria sharing a patient context can be assessed in a batch before clinical verification.Source: §1 · p. 2, Figure 1 · p. 2
• The retrospective evaluation compares JEV with schema-constrained Qwen Flash on 2,048 patient-condition pairs across TrialGPT and question-answer resources derived from MIMIC-III and MIMIC-IV.It assesses conditions individually or in shared-context batches and measures classification, observed job latency, and estimated cost.Source: §1 · p. 2

2.1 Criterion-level assessment

• Each patient-condition pair receives a label describing whether the stated condition holds, does not hold, remains unresolved, or does not apply; INSUFFICIENT concerns available evidence, not a confidence threshold.For an exclusion criterion, MET indicates that the disqualifying condition holds.Source: §2.1 · p. 3, Table 1 · p. 3

2.2 JEV: native decisions and option probabilities

• For each question, JEV returns a selected label, probabilities over permitted labels, and a derived confidence score, but no explanatory rationale.For a fixed label set, confidence is an affine rescaling of the maximum option probability; probability diagnostics use the returned vector and selected-label probability.Source: §2.2 · p. 3

2.3 Qwen: schema-constrained label generation

• Qwen receives the same patient text, policies, and label descriptions through a chat interface, with a strict JSON schema requiring one permitted label per condition.No comparable native class-probability vectors were recorded for Qwen.Source: §2.3 · p. 3

2.4 Single and Batch requests

• Single mode submits each condition separately, whereas Batch submits a job’s questions together using a shared patient context.Both modes produce one label per condition; classification is measured at that level, while job latency covers the condition group.Source: §2.4 · p. 4

3.1 Source tasks and retained conditions

• TrialGPT retained 1,014 conditions; MIMIC-III retained 532 Boolean questions; MIMIC-IV retained 1,500 Boolean questions and derived 89 LVEF conditions for values ≤40%.Of 711 MIMIC-IV exclusions, 700 numerical questions lacked a verified conversion rule; 11 were excluded for a blank answer or out-of-range values.Source: §3.1 · p. 4

3.2 Reference-label mapping and interpretation

• Source references were mapped to standardized assessment labels while preserving each source's policy for interpreting missing information.TrialGPT permits some clinically supported inferences from absent documentation, whereas MIMIC questions retain their specified scope, numerical boundary, and default rules.Source: §3.2 · p. 5

3.3 Grouping and retained partitions

• The collection contains 3,135 conditions across development, primary test, and retained confirmation partitions, with existing partition assignments preserved.Primary analyses use 774 TrialGPT, 322 MIMIC-III, and 952 MIMIC-IV test conditions.Source: §3.3 · p. 5, Table 3 · p. 6
• All 30 MIMIC-IV test references labeled INSUFFICIENT come from the derived LVEF question, limiting those results to that question family.Most MIMIC-III jobs contain one condition, leaving limited opportunity for sharing patient context across questions.Source: §3.3 · p. 6

4.1 Comparison and execution

• Four primary configurations were evaluated: JEV Single, JEV Batch, Qwen Single, and Qwen Batch.Each mode used the same frozen patient text, question-level assessment policies, and label descriptions, rendered through its model interface.Source: §4.1 · p. 6
• Raw responses, status, token usage, and request- and job-level timing were recorded for subsequent analysis.The primary Qwen setup disabled thinking and used a 4,096-token completion limit; a separate batch experiment compared thinking off and on under an 8,192-token limit.Source: §4.1 · p. 7

5.2 Single versus Batch

• Switching JEV from Single to Batch changed correct test decisions by +5, −1, and +1 across the three datasets, while its macro-F1 changes were +0.64, −0.33, and −0.10 points, with all paired intervals including zero.Qwen Batch had +15, +5, and +21 net correct decisions and macro-F1 changes of +1.50, +1.14, and +8.22 points; on MIMIC-IV it corrected 48 Single errors but introduced 27 errors on previously correct conditions.Source: §5.2 · pp. 7–8
• On MIMIC-IV, 13 of Qwen Batch’s corrections were among 30 reference-unknown conditions, with no damages in that class; one saved execution per mode cannot separate packaging effects from repeated-call variability.Source: §5.2 · p. 8

5.3 Thinking under matched settings

• Thinking raised macro-F1 point estimates by 2.39 points on TrialGPT and 2.95 points on MIMIC-III, but lowered it by 2.89 points on MIMIC-IV; all paired intervals include zero.The matched test-only comparison also found complete-job median latency increases of 4.30, 2.33, and 3.08 times and estimated cost ratios of 1.94, 1.20–1.24, and 1.48 across TrialGPT, MIMIC-III, and MIMIC-IV, respectively.Source: §5.3 · pp. 8–9, Table 7 · p. 9

5.1 Overall criterion-level quality

• JEV Single macro-F1 was 0.646/0.771/0.808 versus Qwen Single 0.645/0.523/0.732 across TrialGPT/MIMIC-III/MIMIC-IV; MIMIC-III Batch difference was +23.42 points (95% interval 18.73 to 28.05).Source: §5.1 · p. 7, Table 5 · p. 8, Figure 4 · p. 9

5.4 Cost and job timing

• JEV Batch had lower observed median job latency than Qwen Batch on TrialGPT (1.20 versus 4.55 seconds), MIMIC-III (1.18 versus 2.80 seconds), and MIMIC-IV (1.40 versus 8.90 seconds).Estimated upper-bound costs per 1,000 conditions were USD 0.0170 versus 0.0645, 0.2045 versus 0.4314, and 0.0216 versus 0.0684, respectively; within JEV, Batch timing changed little on MIMIC-III, where 322 test conditions were distributed across 319 jobs.Source: §5.4 · pp. 9, 11, Figure 5 · p. 10

• JEV’s quality advantage is conditional on task and request configuration: its largest JEV–Qwen difference is on MIMIC-III and concentrated in reference-unknown conditions, while Batch macro-F1 point estimates are close on TrialGPT and MIMIC-IV.Source: §6 · p. 11
• JEV’s native choice interface can provide criterion-level decisions and label probabilities at low observed Batch latency and estimated cost.Source: §6 · p. 11
• Additional Qwen thinking increases resource use without a uniform quality gain in the matched comparison.Source: §6 · p. 11

A.1 All frozen partitions

• Native-label macro-F1 is reported by frozen partition, and all partitions were previously inspected; “confirmation” is a historical identifier.Source: Table 8 · p. 12

A.2 Paired model comparison

• The direction-aware three-class TrialGPT endpoint had macro-F1 of 0.6388 for JEV Single, 0.6411 for JEV Batch, 0.6563 for Qwen Single, and 0.6526 for Qwen Batch.It merges insufficient information with failure to pass a condition, so its scores are not a performance change on the native four-class task.Source: §A.1 · p. 12

• Paired model comparisons report JEV-minus-Qwen macro-F1 differences and their 95% paired intervals in percentage points, with net correct defined as corrections minus damages relative to Qwen.Source: Table 9 · p. 12
• Single-to-Batch corrections and damages are paired counts, but these counts do not establish stability under repeated calls.Source: Table 10 · p. 13

A.3 Primary baseline resources

• Table 11 reports estimated upper-bound cost per 1,000 planned conditions and observed timing and latency percentiles under different acquisition conditions.Terminal failures remain included in the timed/planned job counts.Source: Table 11 · p. 13
• The aggregate materials retain exact usage, lower and upper cost bounds, tail latency, technical failure counts, and class metrics.Provider-reported reasoning tokens are included in output-token accounting, not charged a second time.Source: §A.3 · p. 13
• The matched thinking test resource panel is reported separately and does not pool MIMIC test with confirmation.This preserves the separation between the MIMIC test and confirmation materials.Source: §A.3 · p. 13

A.4 Native probabilities and adapted family roles

• Table 12 reports JEV native-probability diagnostics on historical test conditions, with calibration and high-confidence error metrics limited to valid vectors.The caption cautions that these values establish neither clinical risk thresholds nor a cross-model calibration ranking.Source: Table 12 · p. 14
• Table 13 describes inherited MIMIC-IV adapted family roles, not independent clinical adjudications or complete protocol criteria.Source: Table 13 · p. 14
• Table 14 summarizes historical test-split INSUFFICIENT-class errors, with precision, recall, F1, and directional confusion counts defined in its caption.These are benchmark label errors, not abstention decisions.Source: Table 14 · p. 15

A.6 Error Structure and Representative Cases

• The eight purposively selected cases illustrate successes and failures against supplied labels, but do not estimate reasoning-mechanism prevalence.The cases compare the same frozen notes and target criteria across JEV Single and Batch and the historical non-thinking Qwen comparator.Source: §A.6 · p. 15, Table 15 · p. 15
• In C1–C2, JEV preserves uncertainty for an unestablished history and matches the negative reference for an explicitly denied symptom, unlike Qwen.These examples distinguish missing historical evidence from an expressly denied symptom; the outputs do not reveal internal decision processes.Source: §A.6 · pp. 15, 17
• In C4–C6, chronology and multi-component requirements expose disagreements: JEV returns INSUFFICIENT for C4 and matches that label in C6, while Qwen matches C4’s positive reference and returns MET in C6.For C5, JEV Single, JEV Batch, and Qwen Batch return the negative reference, while Qwen Single returns MET; these saved examples do not identify the cause of the numerical error.Source: §A.6 · p. 17
• The C7 and C8 examples show mode and evidence-status disagreements, while one saved execution per mode cannot establish stability under repeated requests or rephrasing.For C7, only JEV Batch matches INSUFFICIENT; for C8, both Qwen modes match NOT_MET while both JEV modes return MET.Source: §A.6 · p. 17

A.7 Frozen input identities

• The audit checked frozen input and reference identities and verified all 32 displayed predictions, while keeping restricted identifiers and clinical text outside the distributed package.The review used stored inputs, labels, and outputs without new model calls or clinician adjudication; examples were selected after outputs were observed.Source: §A.7 · p. 18, §A.6 · p. 18