PaperlayerView exact revision on arXiv

Source-linked AI summary

Can Jev Judge Radiology Reports? Evaluating a System One Model for Clinical Factuality

Jiaju Huang, Hao Yang, Xinyu Ma, Xinglong Liang, Kunyan Cai, Junqiang Ma, Shaobin Chen, Yue Sun, Tao Tan

arXiv:2609.27607v1cs.CLcs.AI

TL;DR

Generated radiology reports can omit, add, or reverse findings, making factual differences important to evaluate. This paper tests Jev as a bidirectional statement-level judge and finds that its one-question configuration retains close expert agreement at lower judgment cost, while identifying boundaries in benchmark agreement and study scope.

  • Problem

    Generated reports can add unsupported findings, omit abnormalities, or reverse their presence, so evaluation must capture disagreements and omissions relative to physician-written references.

    Source: §1 · p. 1

  • Method

    Jev judges statements extracted from generated and reference reports against the opposing report and combines the discrepancies in both directions.

    Source: §3.1 · p. 3, §3.3 · p. 4

  • Results

    OneQ retains expert agreement close to Full while reducing judgment API input cost by 43–45%, and Jev outperforms an open NLI judge under matched decomposition and aggregation.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Jev provides a low-cost way to score collections of generated reports, with individual support judgments available for inspection.

    Source: §6 · p. 10

    The study covers one Jev version, small expert benchmarks, and controlled injected errors; broader generator and case distributions would test generality.

    Source: §6.1 · p. 11

Abstract

from arXiv · show

An AI-generated radiology report can resemble a physician's report while omitting an abnormality, adding an unsupported finding, or reversing its presence. Measuring these factual differences is essential for evaluating report generators. We study Jev, a System One decision model, as a simple, low-cost judge of agreement with physician-written reference reports. Our evaluator checks whether each statement is supported by the other report and combines these judgments in both directions to capture unsupported claims and omissions. A single-question configuration reaches Kendall correlations of 0.573 on RadEvalX and 0.398 on RadEvalExpert with expert error counts, outperforming an open natural language inference judge under matched decomposition and aggregation. One support question per statement retains similar expert agreement to seven while using 43-45% fewer judgment input tokens. At the documented API price, judgments cost under three cents per hundred report pairs, excluding local decomposition. In a separate controlled-error test, Jev detects false negation with an AUROC of 0.977. Local RadMatch achieves stronger agreement on clinically significant errors in both expert datasets and on total errors in the shared RadEvalExpert subset. Finding-count and error-scope analyses show that benchmark agreement reflects report size and error definitions as well as medical error detection. These results support Jev as a practical judgment component for measuring factual differences in generated radiology reports and identify where more elaborate evaluation remains valuable.

1 Introduction

Jev evaluates factual differences by checking report statements against one another in both directions, targeting unsupported claims and omissions. One support question per statement offers a low-cost configuration with expert agreement close to the seven-question design, although finding-count and error-scope analyses qualify what benchmark agreement measures.

1 Introduction

  • Jev extracts statements from both reports, judges each against the other report, and aggregates discrepancies, using expert-annotated reports and controlled errors to evaluate error burden and detection.The design builds on report decomposition and bidirectional entailment, particularly RadFact [2].Source: §1 · p. 2
  • One support question per statement retains expert agreement close to the seven-question design, reduces judgment API input cost by 43–45%, and outperforms an open NLI judge under matched decomposition and aggregation.Judgments cost under three cents per hundred report pairs, excluding local decomposition; RadMatch achieves higher agreement on clinically significant errors in both expert datasets and total errors in the shared RadEvalExpert subset.Source: §1 · p. 2
  • Finding-count correlations can reflect report size because reports with more findings offer more opportunities for counted errors, even without correct identification of each error.Jev exceeds a finding-count baseline in both expert datasets, while separating factual errors from spelling and repetition perturbations changes the relative ranking of Jev and NLI.Source: §1 · p. 2
  • The study evaluates Jev against expert error counts and controlled medical changes, identifies a low-cost single-question configuration, and analyzes report size, error scope, calibration, and repeatability.Source: §1 · p. 2

2 Related Work

Radiology report evaluation spans text similarity, clinical structure, and learned error scores, while report judges compare findings for discrepancies. This paper adopts bidirectional decomposition and entailment and examines Jev’s typed probabilities, error detection, and minimal-question cost.

2 Related Work

  • Report metrics span lexical overlap, contextual similarity, extracted clinical structure, and learned error scores, whose different targets can rank reports differently.Examples include BERTScore [22], RadGraph [11], RaTEScore [23], and RadCliQ; RadEval supplies common implementations and an expert-annotated dataset [18].Source: §2 · p. 2
  • Prior report judges include GREEN [14], bidirectional entailment in RadFact [2], finding-level matching in RadMatch [6], and context-sensitive clinical evaluation in CRIMSON [1] and AtomiMed [17].This work adopts decomposition and entailment while examining Jev as the judgment engine.Source: §2 · p. 2
  • Medical-judge studies examine model and prompt choices, while this work focuses on Jev’s typed probabilities, medical error detection, minimal question set, and what expert-agreement scores reward.The related work also describes VERT [3] and RadSEM [19].Source: §2 · p. 2
  • Finding counts can confound report-level error agreement because reports with more findings offer more opportunities for counted errors; this study compares count baselines with within-examination ordering.Related evaluation approaches include length-controlled AlpacaEval [8], SummEval [9], and RaTE-Eval’s normalization by potential errors [23].Source: §2 · p. 2

3 Jev as a Radiology Report Judge

The evaluator extracts findings from generated and reference reports, asks Jev whether each is supported by the opposing report, and aggregates discrepancies in both directions. The design includes a one-question configuration, fuller question variants, and an open NLI comparison.

3.1 Task and statement units

  • Given a generated report and physician-written reference, the evaluator assesses findings that disagree or are left unaddressed.L1 splits sentences into atomic findings while preserving negation, uncertainty, laterality, severity, measurements, and comparison phrases; reported OneQ uses L1.Source: §3.1 · p. 3

3.2 One question, three outcomes

  • Jev returns support, contradiction, and noncoverage probabilities for a statement against the opposing report; OneQ applies the same question to candidate and reference statements.Here, not addressed means missing textual evidence, and one question per statement–report input still requires multiple judgments per report pair.Source: §3.2 · pp. 3–4

3.3 Bidirectional aggregation

  • The primary score combines candidate-side discrepancies with reference-side noncoverage as a soft discrepancy count over extracted statements; lower scores indicate greater agreement.Soft precision, recall, and F1 are also reported; normalized counts and F1 examine agreement as a proportion of report content.Source: §3.3 · p. 4

3.4 Configurations and open judge

  • Full adds negation, discrepancy-type, and clinical-significance questions, whereas OneQ uses only support; the open DeBERTa NLI judge shares OneQ’s statements and aggregation.Component analyses test sentence-only splitting, shorter criteria, relevance filtering, alternative discrepancy weights, and significance weighting against Full.Source: §3.4 · p. 4

4 Experimental Design

The experiments evaluate agreement with expert report error labels, controlled sentence-level errors, and repeated runs across report datasets and judge baselines. Statistical comparisons use study-level bootstrap intervals, while API cost is based on input tokens and excludes local decomposition.

4.1 Datasets and targets

  • RadEvalX contains 100 IU-Xray report pairs with consensus error counts, while RadEvalExpert contains 624 pairs from 208 studies with three generated candidates per reference.Controlled ReXErr tests use 19,513 sentences after excluding neutral rewrites, with raw Full support probabilities and no atomic decomposition or negation correction.Source: §4.1 · p. 4

4.2 Comparators and implementation

  • Comparators include standard radiology metrics and a finding-count baseline; RadFact and RadMatch are run on the same valid pair subsets for fair comparisons.Repeated-call experiments hold Jev’s decomposition fixed and bypass its judgment cache, while the generative baselines use a local Qwen backend.Source: §4.2 · p. 5

4.3 Statistics and cost

  • Primary agreement uses Kendall’s τb with study-level percentile bootstrap intervals; API cost counts judgment input tokens at $0.042 per million, excluding local decomposition.Controlled-error analyses report AUROC, recall at 0.5, calibration error, and Brier score; repeated runs measure score and ranking consistency.Source: §4.3 · p. 5

5 Results

Jev tracks expert-assessed report errors and detects several controlled medical changes, with a one-question configuration retaining similar agreement to the full design at lower input cost. Results also show that agreement depends on error scope, report size, aggregation, and evaluation setting.

5.1 Report scores track expert-assessed factual differences

  • τb=0.573 (RadEvalX) and 0.398 (RadEvalExpert) for OneQ; RadMatch leads significant errors on RadEvalX (0.491 vs 0.346) and both targets on shared RadEvalExpert.On the shared 200-pair RadEvalExpert subset, RadMatch’s total-error correlation is 0.453 versus OneQ’s 0.322.Source: Table 1 · p. 6, §5.1 · pp. 6–7

5.2 Detecting changes in medical assertions

  • AUROC 0.977 [0.972, 0.982] for false-negation detection; across injected errors NLI scores 0.8969 versus Jev’s 0.8896, but on factual-error types Jev scores 0.9680 versus NLI’s 0.9614.Jev’s typo AUROC is 0.562 and repetition recall is 0.003; the grouping also includes language perturbations that can change medical meaning.Source: §5.2 · p. 7, Table 2 · p. 7

5.3 One question retains agreement at lower cost

  • −0.009 [−0.026, 0.008] and +0.001 [−0.002, 0.004]: OneQ’s total-error correlation changes little versus Full on RadEvalX and RadEvalExpert.OneQ reduces judgment input tokens by 44.56% on RadEvalX and 43.03% on RadEvalExpert; API cost is about 2.3–2.6 cents per hundred pairs, excluding local decomposition. The retained configuration uses L1 statements, one support question, and default count aggregation; filtering raised RadEvalX input cost by 56.1% without observed scoring advantage.Source: §5.3 · p. 7, Figure 2 · p. 8

5.4 Judgment cost in a concrete example

  • Across three constructed findings—contradiction, support, and an unmentioned finding—Jev and GPT-5.6 Luna returned the same classifications in all five runs; mean API costs were $0.00005044 and $0.00018520, respectively.Median completion times were 1.129 s for Jev and 3.569 s for Luna, using three concurrent requests per method.Source: §5.4 · p. 8

5.5 Interpreting agreement beyond finding counts

  • τb gains over finding count were 0.285 [0.157, 0.416] on RadEvalX and 0.137 [0.080, 0.197] on RadEvalExpert; aggregation also changed agreement.On the common RadEvalExpert subset, OneQ and Full score 0.246 versus RadMatch’s 0.043 for within-study total-error ordering, with intervals including zero. Full’s count score is 0.320 versus 0.111 with soft F1 on that subset; on RadEvalX, it is 0.582 versus 0.561, with the paired interval including zero.Source: §5.5 · p. 8, Figure 4 · p. 9

5.6 Calibration makes probabilities useful for thresholding

  • ECE falls from 0.1235 to 0.0077 for Jev after isotonic calibration on the separate holdout; raw Full scores overestimate expert error counts at the upper end.NLI ECE falls from 0.1072 to 0.0063 under the same calibration procedure.Source: §5.6 · p. 9

5.7 Rankings remain stable across repeated judgments

  • Median between-run rank agreement is 0.990 for OneQ and 0.986 for Full across five uncached runs on the same 50 RadEvalExpert pairs.RadMatch’s identical-score share rises from 10% to 82% and rank consistency from 0.662 to 0.965 when temperature changes from one to zero; GREEN scores are identical across all five runs.Source: Table 3 · p. 10, §5.7 · p. 10

6 Discussion

Jev offers an inexpensive way to score report discrepancies, but the preferred evaluator depends on the error target. The discussion also limits what benchmark agreement establishes and identifies boundaries for generalization.

6 Discussion

  • Jev combines support judgments in both directions to score unsupported claims and omissions, with individual judgments available for inspection.Its report scores track expert-assessed error burden.Source: §6 · p. 10
  • OneQ retains expert agreement close to Full while reducing judgment API input cost by 43–45%; additional questions and filtering offer no consistent gain.The discussion presents extracting statements, asking one support question, and aggregating discrepancies in both directions as a practical starting point.Source: §6 · p. 10
  • On RadEvalX, Jev and RadMatch have similar total-error correlations, while RadMatch agrees more closely with significant errors; on shared RadEvalExpert, RadMatch leads on both targets.The discussion identifies the more elaborate evaluator as an option for researchers prioritizing significant errors.Source: §6 · p. 10
  • Discrepancy counts retain report-size information, whereas normalized scores describe the proportion of content in agreement; error scope and intended use affect metric interpretation.Finding-count baselines, shared-sample comparisons, error-scope breakdowns, and within-study candidate ordering offer complementary views.Source: §6 · p. 10
  • Jev's aggregate rankings repeat closely across runs, while GREEN returns identical scores; decoding settings affect repeatability.RadMatch's rank consistency rises from 0.662 to 0.965 when sampling is disabled on the same inputs.Source: §6 · p. 10

6.1 Limitations and future work

  • The evaluation measures agreement with reference text, which may itself contain errors or omissions; image-based correctness and clinical use require separate studies.The study also covers one Jev version, small expert benchmarks, and controlled injected errors, so broader generator and case distributions would test generality.Source: §6.1 · pp. 10–11
  • Report-level category scores do not yet reliably distinguish location, severity, and false-finding errors.Probability thresholds benefit from calibration on intended data, and language-quality assessment requires coverage beyond factual support.Source: §6.1 · p. 11

7 Conclusion

The conclusion presents Jev as a low-cost way to measure factual differences between generated and reference reports. Its support judgments work in both directions, and the single-question setup retains expert agreement close to the seven-question design at lower judgment cost.

7 Conclusion

  • Jev scores what a generated report changes, adds, and leaves out through statement-level support judgments combined in both directions.The resulting report scores track expert-assessed error burden.Source: §7 · p. 11
  • One support question per statement yields expert agreement close to the seven-question design at 43–45% lower judgment cost.Judgment API cost is below three cents per hundred report pairs, excluding local decomposition.Source: §7 · p. 11
  • The conclusion presents this Jev configuration as a low-cost way to evaluate factual differences in generated reports.Controlled-error tests show sensitivity to several medically meaningful changes, and repeated calls produce closely agreeing report rankings.Source: §7 · p. 11

A Data Processing and Supplementary Tasks

The supplementary material documents expert-label handling, dataset sampling, and additional task results. It also reports RaTE-Eval correlations for sentence errors and paragraph quality ratings.

A Data Processing and Supplementary Tasks

  • RadEvalX total-error labels sum significant and insignificant counts across eight categories, and blank category cells are treated as zero.The 100 annotated report IDs join one-to-one to the released 590-row metric file; unlabeled rows are excluded.Source: A Data Processing and Supplementary… · p. 13
  • RadEvalX contains 100 distinct report IDs, and each ID has one candidate, so within-study candidate ranking is unavailable.Report IDs, rather than identical text alone, are retained as the sampling unit.Source: A Data Processing and Supplementary… · p. 13
  • Table A1 compares directed Kendall τb under all-eight-category and six-category RadEvalX labels, with predictions held fixed.The six-category target excludes added and omitted uncertainty.Source: Table A1 · p. 13
  • RadEvalExpert keeps candidates with their shared reference together when resampling, while neutral ReXErr rewrites are excluded from the main binary-error analysis.Unchanged sentences form the negative class in each error-scope comparison.Source: A Data Processing and Supplementary… · p. 13
  • The ReXErr factual-error group includes changes such as false negation, location, severity, and device attributes; grouping is exploratory and edits were not all manually adjudicated.The two groups share unchanged negatives but use disjoint positive examples.Source: A Data Processing and Supplementary… · p. 13
  • On reconstructed RaTE-Eval synonym–opposite comparisons, Full gives synonyms lower error scores in 97.3% and higher soft F1 in 99.9% of comparisons.After excluding seven comparisons below the 0.20 preprocessing-confidence threshold, the rates are 97.5% and 100.0%.Source: A Data Processing and Supplementary… · p. 13
  • On RaTE-Eval, normalized Jev count correlates with sentence error ratios at 0.398 and paragraph quality ratings at 0.424.RadCliQ reaches 0.487 on paragraph ratings, indicating that broader report quality is a distinct evaluation target.Source: A Data Processing and Supplementary… · p. 13, Table A2 · p. 13

B Implementation and Exact Questions

The supplementary methods describe report decomposition, Jev's support judgments, and component analyses. Additional comparisons cover within-study ordering, dataset strata, and exploratory weight settings.

B Implementation and Exact Questions

  • The implementation segments reports, removes template headings and empty units, then uses a local model for atomic decomposition; invalid JSON falls back to the input sentence.The reported OneQ configuration uses L1 atomic decomposition; sentence-level L0 is a separate ablation.Source: B Implementation and Exact Questions · p. 13

B.1 Support question and Full extensions

B.2 Atomic decomposition prompt

  • Atomic decomposition splits each sentence into one finding per anatomical entity while preserving negation, uncertainty, laterality, severity, size, and comparison-to-prior wording.The prompt also instructs the model not to add, infer, or normalize content.Source: §B.2 · p. 14

B.3 Filtering and shortened criteria

C Additional Expert Comparisons

  • Within-study ordering scores whether method and expert orderings agree, disagree, or tie across candidate pairs, with all pairs included in the denominator.The full RadEvalExpert set contributes 624 comparisons; the shared RadFact/RadMatch subset contributes 69 comparisons across 138 studies.Source: C Additional Expert Comparisons · p. 15

C.1 Source, section, and modality

  • Jev–Full has positive total-error correlation across RadEvalExpert data sources, with wider uncertainty for the smallest source.RaTE-Eval performance varies by modality and target, and paragraph quality ratings often favor BERTScore or RadCliQ.Source: §C.1 · p. 15, Table A4 · p. 15, Table A5 · p. 16

D Complete Component Comparisons

  • Exploratory weights peak at τb = 0.635 on RadEvalX with wh = wo = 0.5, and τb = 0.410 on RadEvalExpert with wh = 0.5 and wo = 1.These settings were compared on evaluation data, so the largest observed value is not an independently validated tuning gain.Source: D Complete Component Comparisons · p. 15, Table A6 · p. 16

E Cost and Latency

The section quantifies token savings for the reduced-question configuration and reports the API cost and latency of a three-finding demonstration. It also notes that the workload-cost comparison includes cache hits, while the reported request latencies exclude them.

E Cost and Latency

  • 44.56% fewer input tokens on RadEvalX and 43.03% fewer on RadEvalExpert are reported for the reduced configuration relative to Full.Full and OneQ share the same L1 decomposition, so its local cost is shared; Table A8 excludes local GPU costs.Source: E Cost and Latency · p. 16, Table A8 · p. 17
  • Full's median request latency was 433.6 ms and its 95th-percentile latency was 548.4 ms, but no matched full-pipeline timing comparison is included.Request latencies exclude local cache hits, whereas the workload-cost comparison includes them.Source: E Cost and Latency · p. 16

E.1 Three-finding API demonstration

  • The three-finding demonstration classified all 30 cases correctly across five rounds, with judgment-request timing excluding decomposition.Five rounds used three constructed claims, one request per finding, concurrency three, and no local response cache or warmup.Source: §E.1 · p. 17
  • Jev's five-run completion times were 0.607–1.212 s, versus 2.869–4.301 s for Luna; per-round costs were $0.000050442 and $0.00018520, respectively.The documented price basis is September 22, 2026 standard API pricing; completion time includes communication and service overhead.Source: §E.1 · p. 17

F Error Types and Language Stress Tests

The error-type analyses test Jev on constructed sentence errors and report-level category scores, while the calibration analyses examine raw probabilities and a held-out calibration procedure. The results include strong performance on selected controlled errors, but also expose category-attribution and probability-calibration limitations.

F Error Types and Language Stress Tests

  • 129 of 131 constructed stress-set cases were classified correctly after review, versus 125 before review; the revised set's laterality, direct-negation, and uncertainty cases were all correct.The revisions followed inspection of model outputs, and the small constructed sets are not estimates of clinical sensitivity for naturally occurring errors.Source: F Error Types and Language Stress Tests · p. 17
  • Two constructed cases failed: broad negative language led to a femur-fracture claim being labeled contradicted rather than unaddressed, and a double-negation statement was also mislabeled.The examples illustrate that broad negative wording can blur contradiction and noncoverage.Source: F Error Types and Language Stress Tests · p. 17
  • False-finding scores exceeded the more specific category scores for location errors (0.468 vs. 0.339) and severity errors (0.493 vs. 0.301), limiting report-level category attribution.Figure A1 shows average predicted category-score shares by injected error class, not a finding-matched confusion matrix.Source: F Error Types and Language Stress Tests · p. 17, Figure A1 · p. 18

G Calibration and Repeated Calls

  • On 19,513 raw ReXErr examples, Jev's lowest probability bin predicted 0.0037 error probability despite an observed error fraction of 0.1672.Figure A2 plots raw sentence probabilities in ten equal-width bins and is distinct from the post-hoc calibrated holdout probabilities.Source: G Calibration and Repeated Calls · p. 18, Figure A2 · p. 18
  • Calibration fits used 9,562 development examples and 9,951 holdout examples split by study; the reported results concern the Full sentence configuration and this error mixture.Logistic calibration fits a sigmoid and isotonic regression fits a monotone map; AUROC remained nearly unchanged, while isotonic maps can introduce ties.Source: G Calibration and Repeated Calls · p. 18

G.1 Repeatability

The repeatability analyses compare repeated Jev judgments and local baselines on fixed report-pair samples. They show low variation across Jev runs on RadEvalX, while baseline scoring stability and validation failures vary with inference settings.

G.1 Repeatability

  • 0.00745 for Full and 0.00737 for OneQ were the sample standard deviations of correlation with expert total-error counts across five runs on 50 RadEvalX pairs.The experiment held 615 extracted findings fixed and used five uncached runs per configuration.Source: §G.1 · p. 18, Table A11 · p. 19
  • 0.468 was RadMatch's mean expert correlation at temperature zero, compared with 0.400 at temperature one; Jev OneQ and Full averaged 0.202 and 0.211.The six settings used the same 50 pairs and five repetitions; between-run rank consistency is the median of ten pairwise Kendall correlations.Source: §G.1 · p. 19
  • Temperature zero improved scoring stability, but RadMatch matching-stage validation still triggered fallback in 17 of 250 evaluations, versus nine at temperature one.Repeated scores and output validation measure different aspects of operational reliability.Source: §G.1 · p. 19
Loading 2609.27607v1…