PaperlayerView exact revision on arXiv

Source-linked AI summary

MedSNIP: Building and Benchmarking Snippet-Level Granularity for Medical Fact Verification

Hasan Iqbal, Sarfraz Ahmad, Hyunjae Kim, Sihyeon Park, Junjie Liao, Qingyu Chen, Preslav Nakov, Yuxia Wang

arXiv:2609.12884v1cs.CL

TL;DR

Medical judgments can depend on relationships among clauses and patient context, which atom-level verification may separate. The paper introduces a snippet-level benchmark and generation pipeline, finding that snippet-level verification preserves or improves false-class F1 across three datasets under capable verifiers. The results support treating granularity as a design choice for medically structured claims, not a universal replacement for atom-level verification.

  • Problem

    Medical claims can require linked clinical evidence or patient context, while existing resources evaluate near-atomic claims rather than clinically structured units.

    Source: §1 · p. 1, §2 · p. 3

  • Method

    The paper introduces MEDSNIP-BENCH with human-annotated snippet boundaries and dual correctness labels, and MEDSNIP, a pipeline that groups related clauses into snippets.

    Source: §1 · pp. 1–2

  • Results

    Across MEDSNIP-BENCH, HEALTHFC, and MEDHALLU, snippet-level verification improves false-class F1 under capable verifiers while reducing verifier calls.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Verification granularity is a design choice for medically structured claims, not a fixed preprocessing default or universal replacement for atom-level verification.

    Source: §6 · p. 9, Limitations · p. 10

    The experiments focus on English medical fact-checking; other languages and domains may require different granularity choices.

    Source: Limitations · p. 10

Abstract

from arXiv · show

A medical claim's correctness often depends not on the claim alone, but on the clinical structure around it. A claim may require a lab reference range, a causal or conditional link, or patient-specific details to be judged correctly, and atom-level decomposition can fragment these dependencies, leaving the verifier with clinically incomplete claims. We reformulate medical fact-checking around snippet-level verification, where clause-grouped units preserve local clinical structure. We introduce MedSNIP-Bench, a human-annotated benchmark for snippet-level medical fact verification, and MedSNIP, an automatic snippet-generation pipeline. MedSNIP-Bench covers 276 consumer-health and clinical-vignette responses, segmented into 2,524 snippets with dual in-general and in-patient-context labels and six structural pattern codes. MedSNIP is evaluated against human snippet boundaries on MedSNIP-Bench and then used to generate snippet-level units for external corpora. Across MedSNIP-Bench, HealthFC, and MedHallu, snippet-level verification preserves or improves false-class F1, with gains concentrated where answers are long enough to fragment and where the verifier is strong enough to exploit the recovered structure. The largest merge-pattern gain is on causal-conditional clinical chains. It also reduces verifier calls by 24-73%, though the saving survives end-to-end only when decomposition is cheap, which an open-weight decomposer makes possible at no loss of chunking fidelity.

1 Introduction

Medical fact-checking can fail when atomization separates a clinical judgment from the evidence, interpretation, or patient context needed to assess it. The paper proposes snippet-level verification and introduces a benchmark and pipeline for evaluating it.

1 Introduction

  • Medical claims may depend on linked evidence, reference ranges, causal relations, or patient details, which atomization can split into fragments that are clinically incomplete.A judgment shown in Figure 1 becomes three or four separate claims under atom-level decomposition.Source: §1 · p. 1, Figure 1 · p. 2
  • A snippet groups related clauses into a verifiable unit that preserves the context needed to assess a clinical judgment.Examples include a finding with its interpretation, a lab value with its reference range, or a diagnosis with its caveat.Source: §1 · p. 1
  • MEDSNIP-BENCH provides 276 consumer-health and clinical-vignette responses, 2,524 human-annotated snippet boundaries, two correctness labels, and six structural pattern codes.Source: §1 · p. 2
  • MEDSNIP automatically generates snippets, is evaluated against human snippets on MEDSNIP-BENCH, and supplies snippet-level units for external corpora including HEALTHFC and MEDHALLU.Source: §1 · p. 2
  • The paper evaluates snippet-level verification across models and datasets using paired bootstrap confidence intervals and analyzes performance by structural pattern.Source: §1 · p. 2

2 Related Work

Prior fact-checking work commonly verifies atomic claims, while medical judgments can depend on relationships among clauses. The paper positions snippet-level verification and MEDSNIP-BENCH as a response to that gap.

2 Related Work

  • FACTSCORE and successor systems commonly decompose generations into atomic claims, verify them independently, and aggregate their verdicts (Min et al., 2023; Chern et al., 2025; Wei et al., 2024; Dhuliawala et al., 2024; Iqbal et al., 2024).Source: §2 · p. 2
  • Prior work questions whether the smallest claim is always the best unit: Wanner et al. (2024) show verifier accuracy depends on granularity, and Lu et al. (2025) learn a verifier-preferred atomicity policy.Source: §2 · p. 2
  • Medical resources and methods address claim-level verification, but lack snippet structure and paired general-medical and patient-context labels.The paper describes HEALTHFC, MEDHALLU, MEDSCORE, and MEDFACT as examples of this related work.Source: §2 · p. 3
  • MEDSNIP-BENCH is presented as the first human-annotated snippet-level medical fact-checking dataset, preserving clinical claims with their context.Source: §2 · p. 3

3 MEDSNIP-BENCH

MEDSNIP-BENCH re-segments medical answers into clinically structured snippets and labels them for general and patient-context correctness. Its annotations show substantial boundary agreement and include cases where those two correctness judgments diverge.

3 MEDSNIP-BENCH

  • The benchmark draws on 276 entries—140 consumer-health and 136 clinical-vignette responses—from a parent corpus with 5,755 physician-labeled atomic claims.Source: §3 · p. 3
  • Its six structural patterns distinguish merge cases—enumerations, causal or conditional chains, and conclusions with premises—from standalone facts, topic shifts, and distinct facts that should remain separate.Source: §3 · p. 3

3.2 Dataset Statistics

  • 124 snippets (4.9%) have divergent in-general and with-patient-context labels, concentrated in clinical-vignette entries.Source: §3 · p. 4

3.1 Annotation

  • Annotators grouped atoms using linguistic and logical judgments, revised snippets into self-contained statements, and assigned both general and patient-context correctness labels.They used physician labels inherited from the parent corpus; all snippet boundaries, pattern codes, and correctness labels were assigned manually.Source: §3 · pp. 3–4, §3.1 · p. 4
  • Boundary agreement reached a mean pairwise adjusted Rand index of 0.723; Fleiss’ κ was 0.662 for in-general labels and 0.472 for with-patient-context labels.Source: §3.1 · p. 4

3.2 Dataset Statistics

  • The 5,755 parent-corpus atomic claims map to 2,524 snippets: 1,554 multi-atom and 970 single-atom, with a median of two atoms per snippet.Source: §3.2 · p. 4
  • External evaluation uses 327 HEALTHFC claims and 2,000 MEDHALLU items; because those datasets lack snippet boundaries, MEDSNIP generates snippets before evaluation.The HEALTHFC set excludes 423 insufficient-evidence claims from F1 computation.Source: §3.2 · p. 5

4 Method

The method treats verification granularity, context, and verifier design as separate choices. MEDSNIP generates clause-grouped snippets, which are checked either alone, with the answer context, or with retrieved evidence.

4 Method

  • The comparison varies verification unit, context, and verifier design; atoms are standalone claims, while snippets are grouped clauses.Human snippets are used on MEDSNIP-BENCH; MEDSNIP generates snippets for HEALTHFC and MEDHALLU.Source: §4 · p. 5

4.1 MEDSNIP Pipeline

  • MEDSNIP produces snippets from an answer and optional query, retaining source pointers, structural codes, and shared context.It offers atomize-then-group and snippet-direct modes; the former better matches human boundaries, while snippet-direct achieves higher FF1 across the three datasets.Source: §4.1 · pp. 5–6

4.2 Verification Modes

  • CLAIM-ONLY checks a unit by itself; FULL-CONTEXT adds the original question and answer to disambiguate it without treating the answer as evidence.The latter tests whether context lost during decomposition can be recovered at verification time.Source: §4.2 · p. 6

4.3 Retrieval-Augmented Verifier

  • The retrieval-augmented verifier iteratively searches for evidence, accumulates it, and accepts TRUE or FALSE only above a confidence threshold.It can abstain when evidence is insufficient and runs for up to five retrieval rounds before a final prompt.Source: §4.3 · p. 6

5 Experiments and Results

Across datasets and verifier models, snippet-level verification can improve false-class F1 and reduce verifier calls, but its benefit depends on source structure, verifier capability, and decomposition costs. Pattern and retrieval analyses further identify where gains arise and when abstention trades coverage for FF1.

5.1 Experimental Protocol

  • Evaluation spans three datasets and six verifiers, using claim-only FF1 and paired bootstrap intervals over 10,000 resamples.On the external datasets, OR-FALSE labels an answer FALSE if any unit is predicted FALSE.Source: §5 · p. 6, §5.1 · pp. 6–7

5.2 Snippet Verification Improves FF

  • Human snippets outperform atoms on MEDSNIP-BENCH for every verifier, while external gains from snippet-direct are mixed and grow with verifier strength and source structure.The human-automatic gap depends on the verifier, and weaker models can lose accuracy on richer snippets.Source: Table 2 · p. 7, §5.2 · p. 7

5.3 Structure and Snippet Gains

  • With GPT-5.4, MEDSNIP-BENCH’s 244-word, 11-sentence answers and 3.37 atoms per snippet coincide with the largest gap (+0.114 FF1); HEALTHFC is 98% single-sentence with no gain.MEDHALLU falls between the datasets in structure and improvement; the ordering holds only with a strong verifier.Source: Table 3 · p. 7, §5.3 · p. 8

5.4 Per-Pattern Analysis

  • Pattern B causal-conditional chains gain +0.083 FF1, while Pattern E’s atom advantage is uncertain because its 39-snippet interval crosses zero.For single-atom patterns D and F, boundaries often coincide, so gains may reflect cleaner text rather than grouping.Source: Table 4 · p. 8, §5.4 · p. 8

5.5 Robustness of the Decomposer

  • GPT-OSS-20B decomposition makes snippet verification 34.1% cheaper end to end, whereas GPT-5.4 decomposition costs 19.1% more than atom verification.Across the full matrix, verifier-column means span 0.113 FF1 versus 0.019 for decomposer rows; boundary fidelity and downstream utility differ.Source: Table 5 · p. 8, §5.5 · pp. 8–9

5.6 Retrieval-Augmented Verification Provides a Coverage Lever

  • At τ = 0.90, retrieval reaches 0.609 FF1 at 63% dev coverage and 0.552 at 61% test coverage; full coverage does not beat the single-call baseline.Full-coverage ensembles and subset routers improved dev results but did not replicate on test.Source: Table 6 · p. 9, §5.6 · p. 9

6 Conclusion and Future Work

The paper treats verification granularity as a core design choice, introducing a benchmark and pipeline for snippet-level medical fact-checking. Across three datasets, snippet-level verification improves false-class F1 under capable verifiers, while future work would explore adaptive granularity and broader settings.

6 Conclusion and Future Work

  • MEDSNIP-BENCH provides human-annotated snippet-level medical fact-checking data, while MEDSNIP automatically generates snippet-level verification units.The benchmark includes 276 responses and 2,524 snippets with dual correctness labels and six structural pattern codes.Source: §6 · p. 9
  • Across MEDSNIP-BENCH, HEALTHFC, and MEDHALLU, snippet-level verification improves false-class F1 under capable verifiers, with the strongest gains on causal-conditional clinical chains.The benefit depends on available structure and the verifier’s ability to use it.Source: §6 · p. 9
  • Future work will explore adaptive choices among atomic, snippet-level, and document-level verification and extend MEDSNIP beyond English with richer clinical evidence sources.The proposed settings include guidelines, electronic health record–style data, and longitudinal patient context.Source: §6 · p. 9

Limitations

The reported findings are bounded by dataset scope, verifier and aggregation choices, text-quality effects, sample size, and pipeline design. The authors present snippet-level verification as a choice for medically structured claims, not a universal replacement for atomic verification.

Limitations

  • The experiments cover English medical fact-checking, and the authors frame snippet-level verification as a design choice for medically structured claims rather than a universal replacement for atomic verification.Other languages and domains may require different granularity choices, especially when claims are more independent or retrieval coverage differs.Source: Limitations · p. 10
  • The atom–snippet gap varies with verifier strength, prompt context, and aggregation; weak verifiers in FULL-CONTEXT mode can reduce or reverse the snippet advantage.The main experiments use six verifier models and GPT-5.4 for decomposition across datasets.Source: Limitations · p. 10
  • When snippet and atom boundaries coincide, clearer wording may explain gains; wording explains nearly all keep-atomic gains but only about a quarter of merge gains.Repeated verification of identical text changes FF1 by 0.013, so smaller differences may reflect variation rather than an effect.Source: Limitations · p. 10
  • The sample separates the largest pattern categories but not the smallest, and external evidence is bounded by HEALTHFC’s 327 binary-support claims.Pattern E has 39 snippets and a negative-delta confidence interval spanning [−0.300, +0.095]; Section 5.3 reports an ordering across three corpora rather than a fitted relationship. External-dataset snippets use one LLM decomposer, while the six-model decomposer sweep is limited to MEDSNIP-BENCH.Source: Limitations · p. 10
  • The retrieval-augmented verifier uses general web and biomedical-literature retrieval rather than specialized clinical-evidence sources.The authors note stronger retrieval may improve coverage and absolute FF1, but retrieval design is not the work’s main focus.Source: Limitations · p. 10

Ethics and Broader Impact

The paper frames automated medical fact-checking as evaluation methodology that requires expert oversight, not as medical advice or solved factuality evaluation. Its appendix describes the benchmark data, stratified splits, shared context, and annotation agreement.

Ethics and Broader Impact

  • Automated verification is intended to support auditing and research, not medical advice, diagnosis, treatment, or autonomous clinical decision support, and remains imperfect.The dataset contains no real patient records or identifiable clinical documentation, but may reflect biases in educational materials, English-language evidence, and annotation practices.Source: Ethics and Broader Impact · p. 10

A Data Construction Details

  • MEDSNIP-BENCH construction documents corpus partitioning, train/dev/test splits, inter-annotator agreement, and pipeline fidelity evaluation.The parent corpus combines consumer-health and USMLE-style vignette questions.Source: A Data Construction Details · p. 13, §A.1 · p. 13

A.1 Subset Selection

  • A word-length threshold partitions 276 entries into 140 consumer-health and 136 clinical-vignette cases; vignette responses have a higher atom-level false rate than consumer responses (10.0% vs. 6.5%).Consumer queries skew toward enumeration, while vignettes skew toward causal-conditional chains and conclusions with premises.Source: §A.1 · p. 13

A.2 Train / Dev / Test Split

  • Seed 202 yields the best split balance, with a worst-case spread of 1.1 percentage points, while preserving source-subset proportions and matching snippet-level false rates across splits.The split search stratifies by source subset and a coarse in-general false-rate bucket.Source: §A.2 · pp. 13–14, Figure 4 · p. 14

A.3 Shared Context

  • Each entry’s shared context is a structured, subset-dependent key–value summary of query and response framing that applies across its snippets.The schema includes patient details for vignettes and lighter topic or drug fields for consumer-health entries; it is open to additional keys.Source: §A.3 · p. 14

A.4 Inter-Annotator Agreement

  • On the 64-snippet IAA batch, snippet partitions had mean pairwise ARI 0.723, while in-general labels had Fleiss’ κ 0.662 and in-patient-context labels had Fleiss’ κ 0.472.All six annotators agreed on 52 of 64 in-general items; pattern agreement was κ ≈0.59, with residual confusion especially between enumeration and conclusion-with-premises cases.Source: §A.4 · p. 14

A.5 Auto-Snippet Pipeline Quality

MEDSNIP’s atom-to-snippet mode more closely matches human boundaries than snippet-direct mode, though both under-segment relative to the human benchmark. The appendix also describes label-projection checks and the guideline for merging atoms when separation removes needed context.

A.5 Auto-Snippet Pipeline Quality

  • Mode 1 achieves sentence-set F1 of 0.756 against human snippet boundaries, compared with 0.734 for Mode 2; embedding cosine similarity is 0.824 versus 0.794.The pipeline under-segments: Mode 1 produces 1,578 snippets and Mode 2 2,132, against 2,524 human snippets; consumer entries are easier to reproduce than vignettes.Source: §A.5 · pp. 14–15, Table 7 · p. 15
  • Drop-mixed is the main label-projection policy, while AND and majority serve as robustness checks that agree in sign with it in eleven of twelve verifier-by-mode cells.Projection effects are small relative to model-level differences; dropped cases reflect mixed-label coverage rather than pipeline failures.Source: §A.5 · p. 15

B.1 Background

  • The annotation rule merges atoms when separating them would break a logical dependency or leave too little information to judge medical correctness.If an atom can be checked on its own, it remains atomic; otherwise it is merged with atoms that supply the missing context.Source: §B.1 · p. 15, Table 8 · p. 16

B.2 Task

The annotation workflow defines how clinical context, clause grouping, and dual labels shape snippet construction. The appendices document the prompts, verifier variants, and experimental setup used to implement and evaluate it.

B.2 Task

  • Annotators group related atoms into snippets, record shared context, assign structural codes, and label correctness both generally and for the specific patient or query.Ambiguous labels require notes, and pattern codes identify the structural reason for chunking.Source: §B.2 · p. 16, §B.6 · p. 18

B.3 Shared Context

  • Shared context records query information needed to interpret snippets, including patient details for vignettes and topic or drug details for consumer-health queries.Typical vignette keys include age, sex, medical history, medications, and vitals.Source: §B.3 · p. 16

B.4 Grouping Rules

  • Six structural patterns guide keep-or-merge decisions; genuine hybrids may receive a secondary code, while pattern non-fits go to lead-researcher adjudication.The default is to keep atoms separate unless a merge pattern clearly applies.Source: §B.4 · p. 17, Table 9 · p. 17

B.5 Snippet Text

  • Snippet text must stand alone for verification, resolve references using shared context, and add no medical information beyond the source atoms.Annotators review and edit an LLM-generated draft.Source: §B.5 · p. 17, §B.7 · p. 18, Figure 6 · p. 18

B.6 Dual Labels

  • In-general labels judge medical correctness, while with-context labels judge correctness for the particular patient or query; disagreements and ambiguity require explanatory notes.Labels are True, False, or Ambiguous.Source: §B.6 · p. 18

B.7 Annotation Dashboard

  • The dashboard supports atom grouping, snippet editing, dual labels, pattern codes, and ambiguity notes; it requires every atom to be assigned before export.Prompts are assembled from reusable guideline, schema, formatting, granularity, and task-instruction components.Source: §B.7 · p. 18, C Decomposition and Verification Prompts · p. 18

C Decomposition and Verification Prompts

  • MedSNIP uses extract, cluster, and direct-generation prompts, while the single-call baseline verifies one atom or snippet per call.Mode 1 extracts atoms then clusters them; Mode 2 generates snippets directly.Source: §C.2 · p. 18, §C.1 · p. 18, Table 10 · p. 19

C.3 Retrieval-Augmented Verifier

  • The retrieval verifier iterates through search and judgment prompts, with a force-final fallback; its setup uses Serper and PubMed retrieval and fixed bootstrap and partition seeds.Each item uses one to five retrieval rounds, averaging about 1.9.Source: §C.3 · p. 19, Figure 17 · p. 20, Figure 18 · p. 21

D.2 Bootstrap CI Methodology

The appendix uses paired bootstrap intervals and selects the interval procedure according to whether observations are flat samples or clustered units.

D.2 Bootstrap CI Methodology

  • 95% paired bootstrap intervals use 10,000 replicates, and significance is declared when the interval excludes zero.Each replicate resamples both comparison arms on the same draw.Source: §D.2 · p. 21
  • For flat samples, the analysis uses BCa intervals to correct bias and skewness, which can arise when F1 approaches zero.This is used for per-pattern comparisons in Table 4.Source: §D.2 · p. 21
  • For clustered resampling, percentile intervals resample whole source entries on MEDSNIP-BENCH and whole claims on external corpora.This accounts for dependence among snippets from one answer and differing unit counts between comparison arms.Source: §D.2 · p. 22

D.3 Full Per-Cell Results

The full results detail the call–performance trade-off, sensitivity to aggregation and decomposition choices, and the conditions under which gains persist.

D.3 Full Per-Cell Results

  • Snippet-level verification shifts the call–FF1 trade-off toward fewer verifier calls while preserving or improving false-class F1.Table 11 reports per-split call counts, false-class F1, and cost; Figure 21 visualizes the trade-off.Source: Figure 21 · p. 22, §D.3 · p. 22, Table 11 · p. 23

D.4 Aggregation Rule Sensitivity

  • Under THRESHOLD-k, snippets win in 12 of 14 settings with P(∆>0) ≥ 0.95; under OR-FALSE they win in 7 of 14, while AND-FALSE reverses MEDHALLU by 0.015 FF1.The AND-FALSE rule favors atomization because it requires every unit to be false, and atomization produces more units.Source: §D.4 · p. 22

D.5 End-to-End Cost

  • Snippet-direct reduces total calls by 60.1%, 20.9%, and 17.3% on MEDSNIP-BENCH, MEDHALLU, and HEALTHFC, respectively; GPT-5.4 decomposition makes the full pipeline 19.1% more expensive.GPT-OSS-20B decomposition costs $0.20 and preserves nearly all of the 35.3% verification saving; snippet-direct requires only one decomposition call.Source: §D.5 · p. 22

D.6 Wording Against Grouping

  • Rewriting alone recovers 94% of keep-atomic gains but 26% of merge gains, leaving 74% attributable to grouping; for Pattern B, wording explains 25% of its gain.Pattern E has only 39 units and negative deltas throughout, so the authors do not interpret it further.Source: §D.6 · p. 23

D.7 Full Decomposer-by-Verifier Matrix

  • With GPT-5.4, six decomposers yield gains of +0.032 to +0.114 FF1 over expert atoms; significant losses occur only with LLAMA-3.3-70B.Across verifiers, the decomposer advantage disappears, and Mode 1 and Mode 2 perform similarly overall.Source: Table 14 · p. 24, §D.7 · p. 24

E Retrieval-Augmented Verifier Development Checks

  • Two full-coverage routing variants improved development performance but lacked stable, consistent held-out test gains, so the authors retain confidence-thresholded verification and defer routing and ensembling.Source: E Retrieval-Augmented Verifier… · p. 24
Loading 2609.12884v1…