PaperlayerView exact revision on arXiv

Source-linked AI summary

Grounded Adjudication of Variations across Extracted TimeLines (GAVEL): Comparing Clinical Timelines Against Their Case Reports

Jack Cummins, Sayantan Kumar, Ketan Tamirisa, Jeremy C. Weiss

arXiv:2609.13475v1cs.AI

TL;DR

Clinical timeline evaluations depend on a selected reference and event alignment, both of which can affect evaluation. GAVEL compares two timelines against their case report without choosing either as ground truth; across 126 report-level comparisons, merged timelines were preferred in 77.0% and had fewer discrepancies attributed to them.

  • Problem

    Reference-based scoring treats the selected annotation as correct, while event alignment can also affect timeline evaluation.

    Source: Discussion and Conclusion · p. 9

  • Method

    GAVEL grounds and matches events across two candidate timelines using the case report, then returns a typed verdict and supporting report passage for each discrepancy.

    Source: The GAVEL Adjudication Protocol · pp. 3–4

  • Results

    77.0% of 126 report-level comparisons preferred the merged timeline, whose discrepancies decreased from 7.63 to 0.85 per report.

    Source: Results · pp. 8–9

  • Takeaways & Limitations

    GAVEL supports report-based comparison and revision without treating either timeline as ground truth.

    Source: Discussion and Conclusion · p. 9

    The study is limited to published GLP-1RA case reports, which may themselves omit or wrongly state clinical events.

    Source: Discussion and Conclusion · p. 10

Abstract

from arXiv · show

Existing pipelines for clinical timeline extraction from case reports are evaluated using an expert reference and are limited by imperfect reference annotations and imprecise event alignment. We developed GAVEL, an LLM judge protocol that compares two timelines with the case report and returns a discrepancy type, verdict, and report passage for each difference. We evaluated the event matcher, reviewed 2,738 findings from GPT5.6sol and DeepSeek V3.2, ranked six LLM extractors and two human annotators, and tested GAVEL guided merging. True match rates were 60% immediately below and 48% immediately above the 0.10 cutoff. Manual review confirmed 89.4% and 88.6% of findings. Across 126 reports, merged timelines were preferred in 77.0% of comparisons (95% CI, 69.8 to 84.1%) and reduced discrepancies attributed to the evaluated timeline from 7.63 to 0.85 per report. GAVEL supports report-based comparison and revision without treating either timeline as ground truth.

Related Work

Prior LLM evaluation methods provide pairwise comparison or source-grounded factuality checks, but do not adjudicate event-level differences between two complete extracted clinical timelines. GAVEL applies pairwise judging to those differences using the originating case report.

LLM judges and pairwise evaluation

  • Pairwise LLM judges make comparisons scalable, but typically assess overall response quality rather than whether individual clinical events are supported by the source document.MT-Bench and Chatbot Arena established scalable pairwise judging and Bradley–Terry-style comparison [5,6].Source: Related Work · p. 2

Factuality and clinical evaluation

  • GAVEL matches events across two structured timelines and returns a separate verdict with supporting case-report text for each difference in event inclusion, value, or timing.Prior approaches typically judge responses as a whole, assess one generated output at a time, or score free-text temporal responses.Source: Related Work · p. 2
  • FActScore and VeriFact assess claims in a single generated output, while TIMER evaluates free-text temporal responses rather than disagreements between two timelines.FActScore decomposes text into atomic facts [7], VeriFact checks against patient-specific EHR evidence [8], and TIMER evaluates temporal reasoning over longitudinal records [9].Source: Related Work · p. 2

Study Corpus and Timeline Representation

The study uses a GLP-1RA case-report corpus containing independently constructed clinician timelines and timelines from multiple LLM extractors. Each timeline represents clinical events relative to an index time.

Study Corpus and Timeline Representation

  • The corpus includes independently constructed timelines from two clinically trained annotators and multiple LLM extractors for GLP-1RA case reports.The corpus comes from prior work [4].Source: Study Corpus and Timeline Representation · p. 2
  • Each timeline is a sequence of free-text clinical events paired with times in hours relative to t = 0, the index encounter.Negative and positive times denote events before and after that encounter, respectively.Source: Study Corpus and Timeline Representation · p. 2
  • After excluding six reports with course-critical events available only in unavailable figures, corpus-scale analyses included 126 reports.The study applied the same temporal conventions to LLM-generated and human-annotated timelines; histories without a derivable onset were placed at the encounter.Source: Study Corpus and Timeline Representation · p. 3

Audit of Reference-Based Event Matching

The audit sampled event pairs aligned by the prior cosine-distance matcher and reviewed whether each pair referred to the same clinical event in its report context.

Audit of Reference-Based Event Matching

  • 40,259 unique event pairs assigned finite cosine distances were pooled across the corpus after repeated event-text pairs were removed.The prior procedure recursively aligned candidate and reference events using PubMedBERT cosine distance, retaining alignments below 0.10 [12,2,4].Source: Audit of Reference-Based Event Matching · p. 3
  • 100 pairs were sampled, with 25 from each of four prespecified cosine-distance bands spanning [0, 0.08) through [0.12, 0.30).The two middle bands straddled the operational 0.10 threshold.Source: Audit of Reference-Based Event Matching · p. 3
  • A reviewer examined each sampled pair alongside its source report and labeled whether both event descriptions referred to the same clinical event.Band-specific true-match proportions were reported with Wilson 95% confidence intervals.Source: Audit of Reference-Based Event Matching · p. 3

The GAVEL Adjudication Protocol

GAVEL compares two candidate timelines against the complete case report without designating either timeline as the reference. It returns a typed, report-grounded verdict for each discrepancy.

The GAVEL Adjudication Protocol

  • GAVEL takes the complete report and timelines A and B, then grounds events, matches corresponding events, compares values and times, and classifies discrepancies.A discrepancy is a difference in event inclusion, value, timestamp, or source support identified after events are matched.Source: The GAVEL Adjudication Protocol · p. 3
  • For each event, the judge identifies a supporting report sentence or marks that none was found; matching uses both event identity and report linkage.If neither event appears in the report, events are matched only when they make the same clinical assertion; rows splitting the same event with the same source sentence and timestamp count as one match.Source: The GAVEL Adjudication Protocol · pp. 3–4
  • Table 1 illustrates a timing-only discrepancy: both timelines contain the same event string, and the report states its interval explicitly.The example links the judge’s typed verdict to the report text supporting it.Source: Table 1 · p. 4
  • Times differing by less than max(24 hours, 10% of distance from t = 0) are treated as identical.The 24-hour minimum reflects the study’s day-scale temporal resolution; the proportional term treats small relative differences farther from t = 0 as equivalent.Source: The GAVEL Adjudication Protocol · p. 4
  • Each finding receives one discrepancy type: VALUE, TIMING, A ONLY, B ONLY, SHARED UNSUPPORTED, or DUPLICATE.A matched event differing in both value and time generates separate VALUE and TIMING findings.Source: The GAVEL Adjudication Protocol · p. 4
  • Each finding receives one verdict—A, B, BOTH, NEITHER, or UNCLEAR—and a JSON object with the discrepancy, candidate event details, rationale, and report support.BOTH applies when both representations are report-consistent; UNCLEAR is for differences the report cannot resolve under the temporal conventions.Source: The GAVEL Adjudication Protocol · p. 4

Four-Stage Evaluation Design

The evaluation moves from matcher and judge validation through an eight-source leaderboard and corpus-scale comparison to adjudication-guided timeline merging.

Four-Stage Evaluation Design

  • Stage 1 tested each judge on 30 comparisons and reviewed its findings against the report and both timelines; confirmation measured emitted findings, not missed discrepancies.The judge comparison set covered 10 reports and three extractor pairings.Source: Four-Stage Evaluation Design · p. 4
  • The matcher audit sampled 100 aligned event pairs across four PubMedBERT cosine-distance bands, with 25 pairs per band spanning the 0.10 cutoff.The 0.08–0.12 band region contains approximately 11% of all aligned pairs.Source: Figure 2 · p. 5
  • A second reviewer, blinded to the primary reviewer’s decisions, independently assessed a random, pairing-stratified sample of 150 findings.The sample covered five of the 10 reports.Source: Four-Stage Evaluation Design · p. 5
  • Stage 2 attempted all 28 pairings among eight human and model entrants on each of 10 reports; Stage 1 reports were excluded from this leaderboard sample.The design yielded 280 judge calls.Source: Four-Stage Evaluation Design · p. 5
  • Stage 3 carried forward the highest-rated human, proprietary model, and open-weight model to all 126 usable reports.The three pairings were attempted for every report to test whether leaderboard separation and error profiles persisted at corpus scale.Source: Four-Stage Evaluation Design · p. 5
  • Stage 4 used report-supported edit suggestions to merge timelines for 126 reports, then compared each merged timeline with its original GPT-5 base.A GPT-5.6-sol merger logged whether it applied or skipped each proposed event, timestamp, or value change.Source: Four-Stage Evaluation Design · p. 5

Scoring and Statistical Analysis

The analysis assigns weighted points to source-supported discrepancies, fits pairwise ratings, and normalizes error counts by comparison stage.

Scoring and Statistical Analysis

  • Weighted discrepancy points favored the supported candidate: 3.0 for unsupported additions, 2.0 for wrong times, values, or false duplicates, 1.0 for omitted positive findings, and 0.5 for omitted negatives or recurrences.A candidate’s point share was its weighted total divided by the combined points assigned to both candidates; ties with nonzero totals received 0.5.Source: Scoring and Statistical Analysis · pp. 5–6
  • Table 2 calculates confirmation among findings reviewed by the primary reviewer and groups five specified error categories as “Other.”Those categories include malformed findings, false one-sided findings, wrong placement credited, matching errors, and indeterminate timing.Source: Table 2 · p. 6
  • Timing-weighted and event-weighted alternatives tested whether the Stage 2 rank order depended on the primary discrepancy weights.Judge calls without an A or B verdict were excluded from Bradley–Terry fitting.Source: Scoring and Statistical Analysis · p. 6
  • Bradley–Terry abilities were converted to ratings centered at 1500, with confidence intervals estimated from 2,000 report-level bootstrap resamples preserving within-report pairings.Only differences between entrants within the same analysis are interpretable.Source: Scoring and Statistical Analysis · p. 6
  • Discrepancies were counted as pairwise instances, with rates summed across seven opponents per report in Stage 2, per entrant–report matchup in Stage 3, and per report in Stage 4.UNCLEAR findings were excluded from rankings and error summaries.Source: Scoring and Statistical Analysis · p. 6

Results

Results show substantial uncertainty around the event-matcher cutoff, similar confirmation rates for the two GAVEL judges, and a preference for adjudication-guided merged timelines over the GPT-5 base.

Results

  • At the 0.10 matcher cutoff, true matches were 60.0% immediately below and 48.0% immediately above, with substantially overlapping intervals.Together, the two middle bands contained approximately 11% of 40,259 aligned pairs.Source: Results · p. 6
  • Manual review confirmed 1,376 of 1,540 GPT-5.6-sol findings (89.4%) and 1,061 of 1,198 DeepSeek V3.2 findings (88.6%).Discrepancy coverage was not measured.Source: Results · p. 6
  • Timing disagreements incorrectly assigned UNCLEAR accounted for 126 of 301 errors (41.9%); with two other error types, they made up 77.7%.Remaining reviewer disagreements concentrated in contested anchors and duplicate calls.Source: Results · pp. 6–7
  • In the eight-source round robin, GPT-5, Claude Opus 5, and o4-mini formed a leading tier, with ratings of 1698, 1672, and 1610 and overlapping adjacent intervals.The annotators ranked fourth and seventh, respectively.Source: Figure 3 · p. 7, Results · p. 7
  • Missed positive events (6.1 per case) and wrong times (4.8) dominated Stage 2 errors; GPT-5 had the lowest total error rate (6.0).Claude Opus 5 and o4-mini minimized timing errors but missed more supported events.Source: Results · p. 7
  • Across 126 reports, GPT-5’s rating of 1597 had separate bootstrap intervals from GLM-5.2 (1466) and Annotator 1 (1437); 4,583 findings (31.6%) favored neither candidate.Stage 3 produced 14,493 findings.Source: Results · p. 7
  • GPT-5 had the lowest corpus-scale total error rate (8.1 vs. 16.7 for GLM-5.2 and 14.6 for A1); GLM-5.2 had fewer wrong-time findings but more missed positives than A1.GLM-5.2 had 2.6 vs. 5.5 wrong-time findings and 11.5 vs. 5.5 missed positive events per entrant–report matchup.Source: Results · pp. 7–8
  • 77.0% of 126 report-level comparisons preferred the merged timeline; discrepancies fell from 7.63 per report for GPT-5 to 0.85 for the merged timeline.The merger applied 1,800 of 2,038 suggested edits (88.3%).Source: Results · pp. 8–9, Table 4 · p. 9

Discussion and Conclusion

The discussion shows how report-based adjudication exposes reference and alignment effects, while error profiles distinguish improvement targets. It also notes judge-dependent findings and limits on the study’s scope.

Discussion and Conclusion

  • GAVEL compares timelines against the case report, avoiding selection of either timeline as reference and linking each verdict to report evidence.This lets reviewers trace pairwise scores to individual discrepancies.Source: Discussion and Conclusion · p. 9
  • A1 and A2 had nonoverlapping Bradley–Terry rating intervals, and rankings may change with discrepancy definitions, timing tolerances, or scoring weights.The annotators were not interchangeable reference standards on this corpus.Source: Discussion and Conclusion · p. 9
  • GLM-5.2 and A1 had similar overall ratings but distinct error profiles, separating timing accuracy from supported-event recovery.GPT-5 performed well by balancing event recovery and temporal placement; omissions and timing errors were the main improvement opportunities in this corpus.Source: Discussion and Conclusion · p. 9
  • Two judge models had similar confirmation rates among emitted findings, but differed in output volume and error distributions, affecting which discrepancies were identified.Frequent UNCLEAR verdicts on timing differences considered resolvable by the primary reviewer point to a target for refining adjudication rules and focused clinical review.Source: Discussion and Conclusion · p. 9
  • GAVEL findings served as targeted merge edits, with the largest improvements in event coverage and temporal placement; DeepSeek V3.2 made the final comparison.The merge result therefore remained an LLM-based assessment, not an independent clinical review.Source: Discussion and Conclusion · p. 10
  • The study covers published GLP1-RA case reports, and validation estimates correctness among reported findings, not discrepancy recall.The 10-report leaderboard has rankings that may depend on judge model, candidate order, timing tolerance, and non-decisive verdicts; pairwise adjudication also cannot detect errors shared by both timelines.Source: Discussion and Conclusion · p. 10
Loading 2609.13475v1…