PaperlayerView exact revision on arXiv

Source-linked AI summary

VERGE: Verification-Enhanced Refinement for Grounded Extraction of Early-Onset Colorectal Cancer Symptoms in Clinical Notes

Nikkie Hooman, Monarch Nigam, Amy E. Hughes, Rasmi G. Nair, Mehak Gupta

arXiv:2609.04366v1cs.CL

TL;DR

Early-onset colorectal cancer symptoms in younger adults are common and nonspecific, while follow-up guidance and encounter codes leave gaps in symptom-level detail. VERGE uses evidence-linked extraction and bounded verification to check and revise claims. Across evaluated configurations, it achieved the best precision-recall balance without a corresponding loss in sensitivity, while misattributed evidence remained a key error concern.

  • Problem

    Red-flag symptoms in younger adults are common and nonspecific, follow-up intervals lack evidence-based guidance, and note-level symptom details may not appear in encounter codes.

    Source: §I · p. 1

  • Method

    VERGE extracts six red-flag symptoms and family-history risk status using note-grounded evidence, then applies bounded verification and refinement with an auditable evidence record.

    Source: §I · pp. 1–2

  • Results

    Across development, baseline, and cross-model comparisons, VERGE achieved the best precision-recall balance of evaluated configurations without a corresponding loss in sensitivity.

    Source: §VI · p. 7, §VII · p. 8

  • Takeaways & Limitations

    The dominant error was misattributed rather than fabricated evidence, supporting verification that checks clinical validity alongside textual grounding.

    Source: §VI · p. 7, §VII · p. 8

    The finding list, terminology, and evidence rules are EOCRC-specific, and family-history results are limited by only 18 positive cases and single-clinician labels.

    Source: §VI · p. 8

Abstract

from arXiv · show

Early-onset colorectal cancer is increasing among younger adults, yet red-flag symptoms in this age group have no evidence-based guidelines for follow-up testing, and structured encounter data do not capture the detail needed to support early detection and inform follow-up, including symptom duration, context, and fam- ily history, an established colorectal-cancer risk factor. This study aimed to develop and evaluate an automated method for extracting six red-flag symptoms and family-history risk status from free-text clinical notes. We developed VERGE, an agentic workflow in which an initial label and evidence are proposed using retrieval-augmented generation, then passed through a bounded verification- refinement cycle that checks textual grounding and clinical validity, corrects and rechecks a claim until resolved or a limit is reached, and escalates unresolved claims for human review. VERGE was evaluated on 4,033 clinician-labeled note-finding pairs against a single-agent baseline, a rule-based clinical language-processing baseline, and an alternative underlying language model. Compared with the single-agent baseline, VERGE reduced false positive find- ings, improving precision from 0.764 to 0.849 and MCC from 0.681 to 0.730, a balanced gain across the precision-recall trade-off, and resolved most flagged errors autonomously, with human review required for only 1.5 percent of claims. These results indicate that a bounded, verification-based workflow can reduce unnecessary positive findings without sacrificing the ability to detect true ones. This approach offers a path toward more reliable and trustworthy clinical language-processing tools to support colorectal cancer risk assessment in younger patients.

I. INTRODUCTION

Early-onset colorectal cancer symptoms are common but nonspecific, and existing encounter codes and follow-up guidance do not capture the note-level detail needed to characterize risk and presentation. VERGE combines evidence-linked extraction with bounded verification, building on clinical NLP and multi-stage reasoning approaches while emphasizing whether evidence supports each claim.

I. INTRODUCTION

  • Symptoms are common and nonspecific, follow-up testing lacks evidence-based interval guidance, and encounter codes often miss symptom-level detail and family history found in notes.The introduction frames reliable identification of patients who may not need workup as operationally important alongside identifying those who do.Source: §I · p. 1
  • VERGE extracts six red-flag symptoms and family-history risk status, proposing note-grounded evidence and checking claims for textual grounding, clinical context, and target-specific support.A bounded Verifier–Refiner loop revises claims and escalates unresolved cases for human review.Source: §I · pp. 1–2
  • The study combines retrieval-augmented generation and GRADE-informed evidence prioritization with narrative-over-ROS conflict handling and a bounded, auditable verification workflow.It evaluates the system against baselines and development-stage comparisons and analyzes verification and refinement.Source: §I · p. 2

II. RELATED WORK

  • Prior clinical NLP spans concept-mapping and context-sensitive methods,,, and biomedical language models,, while colorectal-note work has examined symptoms and presentation,.Prior retrieval approaches support terminology coverage or downstream processing,,, whereas the paper identifies verification of note-grounded evidence as a gap in colorectal symptom extraction.Source: §II · p. 2
  • Multi-agent clinical systems and debate illustrate role specialization and repeated checking,,, which VERGE applies in a bounded workflow that preserves traceability and clinical oversight.Source: §II · p. 2

III. DATASET

The study uses retrospective electronic health records from Parkland Health, sampling symptom-stratified encounters for clinician annotation. The resulting analysis set contains 4,033 labeled note–finding pairs from 578 notes and 326 patients.

III. DATASET

  • The source cohort comprises primary-care patients aged 18–49 reporting at least one of six red-flag symptoms at Parkland Health from 2010 through 2020.Patients with prior colorectal cancer diagnosis, colectomy, or gastrointestinal surgery were excluded.Source: §III · p. 2
  • 4,033 labeled note–finding pairs remained after 48 pairs with missing gold labels were excluded, representing 578 notes from 326 patients.A clinician reviewed notes for six symptoms and family history; 583 notes from 327 patients had been available for annotation.Source: §III · p. 2

IV. METHODOLOGY

VERGE frames the task as predicting whether each of seven findings is documented in a clinical note, with positive predictions requiring evidence in that note. Its four-agent workflow proposes and standardizes claims, then verifies and revises them within a bounded loop.

IV. METHODOLOGY

B. Agent 1: Extractor

The Extractor jointly predicts all seven findings from one note, using terminology matching, finding-conditioned retrieval, evidence prioritization, and prompt rules to produce structured, note-grounded outputs.

B. Agent 1: Extractor

  • The Extractor makes one joint prediction for all seven findings from a note, after terminology matching, retrieval, and evidence-quality prioritization.Source: B. Agent 1: Extractor · p. 3
  • Finding-specific dictionaries combine UMLS synonyms and related concepts with manually specified abbreviations to identify relevant note expressions.The finding name is used to obtain a UMLS Concept Unique Identifier before expansion.Source: B. Agent 1: Extractor · p. 3
  • For each finding, three MedCPT query variants retrieve and fuse PubMed passages, with up to 10 candidates proceeding to prioritization.The variants use general colorectal-cancer context, note-detected aliases and synonyms, and broader UMLS terminology; at most two passages per finding are retained by GRADE certainty, with RRF breaking ties.Source: B. Agent 1: Extractor · p. 3
  • Prompt rules prioritize narrative evidence over a conflicting templated ROS negation, and positive predictions must be supported by text in the note.The conflict-resolution rule was developed with a practicing general internal medicine physician at Parkland.Source: B. Agent 1: Extractor · pp. 3–4
  • For each finding, the Extractor returns a binary label, confidence score, note-linked evidence, note section, experiencer, and assertion status.Family-history output additionally requires an explicit biological relation and CRC specificity.Source: B. Agent 1: Extractor · p. 4

C. Agent 2: Claim Composer

The Claim Composer standardizes the Extractor’s outputs into verifiable claims, which the Verifier checks against the full note for grounding and clinical validity. Positive and negative claims follow distinct verification rules before being accepted or sent for refinement.

C. Agent 2: Claim Composer

  • The Claim Composer creates seven structured claims per note, preserving prediction and confidence while organizing evidence and contextual fields for verification.If inference text is missing or malformed, it is retained as evidence; the Composer cannot revise labels or add evidence because it has no note access.Source: C. Agent 2: Claim Composer · p. 4

D. Agent 3: Verifier

  • The Verifier separately checks whether evidence is grounded in the note and whether its assertion, context, and experiencer support the target finding.Family-history claims also require a biological relation and CRC specificity.Source: D. Agent 3: Verifier · p. 4
  • A negative claim is sent for refinement only if note text affirmatively supports the finding, rather than merely mentioning it.Negated, old or resolved, and unconfirmed mentions do not count as affirmative evidence; family-history matches also require the correct relation and cancer type.Source: D. Agent 3: Verifier · p. 4
  • Positive claims require strong or moderate grounding and a clinical-validity pass; negative claims are accepted unless grounded affirmative evidence is recovered.Claims that fail these routing rules are passed to the Refiner with issue tags and the full note.Source: D. Agent 3: Verifier · p. 4

E. Agent 4: Refiner

The Refiner proposes and validates corrections for claims flagged by the Verifier. The bounded loop rechecks changed labels and escalates unresolved claims after five rounds.

E. Agent 4: Refiner

  • Corrections are checked against grounding and clinical context before replacing a claim, and every corrected claim returns to the Verifier.A corrected Yes needs a note-grounded quotation; family-history Yes claims also require a biological relation and cancer-type specificity.Source: E. Agent 4: Refiner · p. 4

F. Verifier-Refiner Loop

  • A claim whose label changes is re-verified, while an unchanged label exits as the retained claim.A label-stable claim exits the loop; a changed label can return to the Refiner if flagged again.Source: F. Verifier-Refiner Loop · p. 4
  • The loop stops after five rounds; unresolved claims retain their current state and are flagged for human review.A deterministic controller records each round’s trajectory and enforces stopping and escalation rules.Source: F. Verifier-Refiner Loop · p. 5

V. EXPERIMENTAL DESIGN AND RESULTS

On 4,033 labeled note-finding pairs, VERGE’s verification and refinement improved precision and F1 over the single-agent baseline while maintaining similar recall. The study also compared extractor configurations, a clinical NLP baseline, and an alternative language-model backbone.

A. Implementation Details

  • Language-model stages used Meta-Llama-3.1-8B-Instruct with greedy decoding, and clinical notes were preserved without truncation.Notes had a median of 383 Llama tokens, with a range of 2–5,110.Source: A. Implementation Details · p. 5

B. Performance Comparison Analysis

  • All configurations were scored on the same 4,033 labeled pairs using precision, recall, F1, and MCC.MCC summarizes performance using all four confusion-matrix outcomes.Source: B. Performance Comparison Analysis · p. 5
  • ROS conflict handling raised recall by correcting false negatives from templated negations but lowered precision; the finalized Extractor achieved the strongest balance among single-agent variants.The finalized Extractor added GRADE-informed retrieval prioritization and served as VERGE’s frozen input.Source: B. Performance Comparison Analysis · p. 5
  • The medspaCy + ConText rule-based baseline had high precision but comparatively low recall, reflecting a conservative operating point.It combines terminology matching with contextual assertion handling.Source: B. Performance Comparison Analysis · p. 5
  • Precision rose from 0.764 to 0.849 and F1 from 0.733 to 0.769, while recall remained similar at 0.705 versus 0.702, with VERGE achieving the highest F1 and MCC among evaluated configurations.The authors attribute the precision gain to fewer false positives and describe the operating point as a better-balanced trade-off.Source: B. Performance Comparison Analysis · p. 5
  • Using Ministral-3-14B-Instruct-2512-BF16 as the agent backbone showed that the verification-and-refinement workflow remained applicable across two model families.The authors report that the benefit was not specific to one backbone.Source: B. Performance Comparison Analysis · p. 5

C. Per-Finding Performance

VERGE’s per-finding gains were largest for rectal pain and rectal bleeding, while family-history results were volatile. Across findings, label changes more often corrected than invalidated labels, and most claims entering refinement resolved within the five-round limit.

C. Per-Finding Performance

  • VERGE’s largest per-finding gains were for rectal pain and rectal bleeding, driven almost entirely by improved precision with recall essentially unchanged.Table III reports per-finding performance and label-change outcomes from Extractor to VERGE; GS+ denotes a correction to the gold standard and GS− an invalidation of a previously correct label.Source: C. Per-Finding Performance · p. 5, Table III · p. 6

D. Verification and Refinement Outcomes

  • Of 367 claims entering refinement, 80 resolved with changed labels and 226 with unchanged labels; 61 claims, or 1.5% of all processed claims, were flagged for human review.The 61 comprised 59 unresolved after label oscillation through the five-round bound and 2 refinement-stage operational errors; Table IV summarizes these outcomes.Source: Table IV · p. 6, D. Verification and Refinement Outcomes · p. 6

C. Per-Finding Performance

  • Family-history performance shifted in both directions because its lower prevalence makes a handful of label changes have an outsized effect.Across findings, label changes were more likely to correct than invalidate labels, largely by removing false positives; the authors link this pattern to verification targeting unsupported positives while missed positives rely on a narrower search.Source: C. Per-Finding Performance · p. 6

E. Nature of Verification Failures

Verification failures more often involved genuine note evidence attached to the wrong clinical claim than fabricated evidence. The exploratory entropy review found few false negatives among flagged negatives and modest recovery with metric trade-offs.

E. Nature of Verification Failures

  • Clinical-validity failures substantially outnumbered grounding failures; some flagged negative claims also recovered missed evidence, revealing both misattribution and omission errors.The findings support checking whether evidence supports the target claim, not only whether it appears in the note.Source: E. Nature of Verification Failures · p. 6
  • In Fig. 2, rectal-bleeding evidence was incorrectly assigned to abdominal pain; verification prompted correction to No, which re-verification confirmed against the gold-standard label.The quoted evidence was grounded in the note but supported a different target finding.Source: E. Nature of Verification Failures · p. 6, Figure 2 · p. 7

F. Exploratory Recovery of Missed Positive Claims via Generation-Time Uncertainty

  • The exploratory method compared generated Yes/No decisions with alternative continuations at their divergence point, using label preference and binary entropy to flag negative claims for review.Elevated-entropy claims were reviewed for missed evidence.Source: F. Exploratory Recovery of Missed… · p. 6
  • Only 53 of 571 high-entropy negative claims (9.3%) were false negatives; review recovered 5 but introduced 35 new false positives.Recall rose slightly (0.702 to 0.709), while precision, F1, and MCC fell to 0.802, 0.753, and 0.706.Source: F. Exploratory Recovery of Missed… · pp. 6–7
  • Flagged false negatives reflected conflicting narrative statements, such as symptoms later reported as resolved, or missed opportunities to search for valid alternative evidence after rejecting a citation.The first pattern differs from the templated-versus-narrative conflict handled by the Extractor.Source: F. Exploratory Recovery of Missed… · p. 7

G. Statistical Comparison

Patient-cluster bootstrap intervals support VERGE’s precision, F1, and MCC gains over Extractor, while recall shows no meaningful change. The discussion highlights a better-balanced precision-recall trade-off, misattributed evidence as the dominant caught error, and limits on scope and validation.

G. Statistical Comparison

  • The 95% confidence-interval differences favor VERGE for precision, F1, and MCC, while the recall interval spans zero, consistent with no meaningful change.The patient-cluster bootstrap keeps each patient’s note–finding pairs together when estimating the paired VERGE-minus-Extractor difference.Source: G. Statistical Comparison · p. 7, Table VI · p. 7

VI. DISCUSSION

  • Across evaluated comparisons, VERGE achieves a better-balanced precision-recall trade-off without becoming meaningfully less able to detect true findings.The discussion links reduced false positives to fewer patients being unnecessarily flagged for further evaluation.Source: §VI · p. 7
  • VERGE’s dominant caught error was misattribution: real, well-quoted evidence applied to the wrong finding.Checking citation relevance to the specific claim matters because real evidence can still support a clinically wrong label.Source: §VI · p. 7
  • Generation-time uncertainty tracked source-text ambiguity but was too imprecise to drive automated correction alone, though it could prioritize claims for secondary review.The discussion proposes deeper reasoning over note structure as a future direction, requiring careful architectural changes and clinical collaboration.Source: §VI · p. 7
  • The results reflect retrospective note abstraction, not validation as an autonomous screening tool.The paper describes VERGE as operating within human-defined bounds, with supporting evidence available for clinical review.Source: §VI · p. 7
  • The workflow’s grounding/validity separation may generalize, but the finding list, terminology, and evidence rules are EOCRC-specific.Labels came from one clinician, and family-history results relied on only 18 positive cases, limiting confidence in that result.Source: §VI · p. 8

VII. CONCLUSION

VERGE pairs bounded autonomous verification with an auditable evidence record for extracting EOCRC symptoms and family-history risk status. The conclusion reports the best evaluated precision-recall balance without a corresponding sensitivity loss, while noting the single-institution, single-clinician cohort.

VII. CONCLUSION

  • Across development, baseline, and cross-model comparisons, VERGE achieved the best precision-recall balance of any evaluated configuration without a corresponding loss in sensitivity.The conclusion connects this pattern to identifying which patients do and do not warrant further workup.Source: §VII · p. 8
  • The dominant failure was misattributed, rather than fabricated, evidence, supporting checks of clinical validity alongside textual grounding.Source: §VII · p. 8
  • The work used a single-institution retrospective cohort with gold-standard labels from one clinician.The authors frame VERGE as a step toward systems whose autonomy stays within human-defined bounds and whose evidence remains available for review.Source: §VII · p. 8
Loading 2609.04366v1…