PaperlayerView exact revision on arXiv

Source-linked AI summary

Recognizing Study Lineage Is Not Enough: Over-Crediting Same-Trial Evidence in LLM-Based Medical Verification

Mukand Krishna, Anoushka Harit, Chenghao Xiao, Zhongtian Sun

arXiv:2610.23721v1cs.CL

TL;DR

Medical evidence may appear to come from multiple publications while tracing to one trial, and it is unclear whether verifiers respect that dependence when aggregating evidence. This paper tests lineage recognition and aggregation separately across source-audited clinical families. It finds that models can recognize independent trial lineage yet still over-credit a same-trial publication.

  • Problem

    It is unclear whether medical verifiers respect trial-level dependence when aggregating distinct publications, even when they can identify shared lineage directly.

    Source: §1 · pp. 1–2

  • Method

    The evaluation separates lineage recognition from evidence aggregation across source-audited clinical families and tests whether study grouping reduces same-trial over-credit.

    Source: §1 · p. 2

  • Results

    Across the original provenance families, GPT-5.5 and selected Qwen3-4B and Ministral-3-8B conditions recognize lineage correctly yet over-credit same-trial reports in separate aggregation calls.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    The results support using underlying study identity, rather than publication count, as the unit of corroboration in these audited families.

    Source: §7 · p. 8

    The 14 selected clinical contrasts are not representative of medical claims or publications, and repeated calls measure within-input stability rather than independent cases.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Publication count is not study count. A model can identify shared trial lineage correctly in a direct task yet over-credit a second publication from the same trial when aggregating evidence. We evaluate this behavior in medical LLM verification using 14 source-audited clinical families and five model conditions. The families span provenance, clinical applicability, result direction, and composition; the provenance families provide the direct lineage tests. Recognition and aggregation are separate stateless calls, and repeated calls assess within-input stability. Across all three original provenance families, GPT-5.5 identifies the independent update in 10/10 recognition calls per family but over-credits the same-trial report in 5/5 separate aggregation calls per family. Selected Qwen3-4B and Ministral-3-8B conditions show the same cross-task contrast. A study-grouping instruction reduces GPT-5.5 over-credit from 4/5 to 0/5 in each later pure-provenance family. Other model conditions show mixed behavior, and rejected outputs limit several comparisons. The results identify a behavioral gap between elicited lineage recognition and evidence aggregation. The code and data artifacts are available at https://anonymous.4open.science/r/study-lineage-review-artifact.

1 Introduction

A verifier may correctly recognize shared trial lineage yet still over-credit a same-trial publication when aggregating evidence. This paper tests that gap across source-audited clinical families and examines whether study grouping reduces it.

1 Introduction

  • Multiple publications may derive from one clinical trial, so treating documents as independent studies can overstate the independence of evidence.Clinical datasets can retain links between trial results and supporting publications, as in CliniFact.Source: §1 · p. 1
  • Prior work examines clinical claim support, source dependence, and whether recognized relations guide later judgments, but does not test trial-level dependence across publications in LLM verification.The paper motivates this source-level test from multi-document repetition findings and work separating relation detection from resolution.Source: §1 · pp. 1–2
  • The evaluation separates lineage recognition from evidence aggregation, treating clinical families as experimental units and repeated calls as within-input stability checks.It covers 14 source-audited clinical families across five model conditions, including provenance, applicability, result direction, and their combinations.Source: §1 · p. 2
  • The central question is whether correct lineage recognition carries over to separate aggregation judgments and whether study grouping reduces over-credit while preserving independent-trial support.Recognition and aggregation are assessed in separate stateless calls.Source: §1 · p. 2
  • Across the original provenance families, GPT-5.5 recognizes the independent trial yet over-credits a same-trial report, a pattern also seen in selected Qwen3-4B and Ministral-3-8B conditions.Study grouping removes GPT-5.5's error in both later pure-provenance families while preserving support for the independent trial; effects vary across other models, and rejected outputs leave some comparisons unresolved.Source: §1 · p. 2

2 Related work

Related work addresses medical claim support, source linkage, repetition, and gaps between recognizing a relation and using it. The paper's evaluation targets whether an LLM verifier respects trial-level dependence when combining distinct publications.

2.1 Medical evidence verification and contextual grounding

2.2 Source dependence and recognition–use gaps

  • Clinical synthesis treats the study, rather than its reports, as the unit of interest, while linkage tools address identifying publications from one trial.The Cochrane Handbook calls for linking multiple reports of one study while retaining useful information from each (Li et al., 2024); Aggregator clusters articles likely to report the same registered trial.Source: §2.2 · pp. 2–3
  • Multi-document studies report that repetition can increase persuasiveness or reverse source preferences, whereas this paper examines distinct clinical publications that may share one trial.Prior work includes GroupQA (Naphade, 2026), Whose Facts Win? (Schuster et al., 2026), and Bara's distinction between report multiplicity and evidential ancestry (Bara, 2026).Source: §2.2 · p. 3
  • Prior evaluations separate relation detection from resolution or test whether verbalized distinctions affect predictions, motivating separate lineage-recognition and aggregation tasks.The two tasks in this paper use independent model calls; adjacent medical-AI work also studies behavioral responses to source relations.Source: §2.2 · p. 3

3 Task formulation

The task models a fixed claim receiving source-linked evidence updates and distinguishes study ancestry, clinical applicability, and result direction. Separate recognition and aggregation calls test whether lineage judgments guide combined-evidence judgments.

3.1 Evidence-update setting

  • Each update to a fixed claim isolates provenance, applicability, or result direction, with independent corroboration requiring both study independence and claim-relevant support.Same-study reports can add relevant detail without adding an independent study; out-of-scope positive evidence leaves the scoped judgment unchanged, while applicable opposing evidence registers conflict.Source: §3.1 · p. 3
  • A later analysis from a study already represented leaves the study count unchanged, while an independent study counts as corroboration only if its result applies and agrees with the claim.Two supportive passages can warrant a supported label without warranting an independent-corroboration flag.Source: §3.1 · p. 3
  • Held-out H3 combines a same-lineage WOSCOPS analysis, out-of-scope positive 4S evidence, and a challenging ALLHAT-LLT result around one statin claim.The source identities are recorded in Appendix Table 4 (Shepherd et al., 1995; Vallejo-Vaz et al., 2017; Scandinavian Simvastatin Survival Study Group, 1994; ALLHAT Officers and Coordinators for the ALLHAT Collaborative Research Group, 2002).Source: §3.1 · p. 3

3.2 Lineage recognition and evidence aggregation

  • Recognition presents two candidate updates and asks which meets one target relation, while aggregation returns a combined judgment without the expected relation or recognition response.Calls are stateless, and candidate order is tested both ways.Source: §3.2 · p. 4
  • Five fresh invocations are run per exact input; invalid response-contract outputs are rejected and receive no semantic score.Relation recognition and relation-specific aggregation are primary outcomes; numerical support and confidence scores are diagnostic.Source: §3.2 · p. 4

3.3 Evaluation outcomes and validity

  • Same-trial over-credit is a valid aggregation output marking a report as independent corroboration despite an audited link to a study already represented.Independent-trial preservation is audited separately and requires both the corroboration flag and use of the added evidence.Source: §3.3 · p. 4
  • The primary over-credit measure is the combination flag, not a change in the 0–100 support score.A report may add detail and affect assessment without becoming a second independent study.Source: §3.3 · p. 4

4 Benchmark construction

The benchmark uses source-audited clinical families with fixed claims and matched updates that isolate declared relations. The staged suites distinguish the original evaluation from follow-up and diagnostic analyses.

4.1 Source-audited clinical families

  • Source audits retained exact passages and locators, checked trial lineage, assessed clinical scope, and assigned result direction from claim-relevant findings.Statistical significance served as supplementary context.Source: §4.1 · p. 4
  • Families were accepted only when initial evidence supported assessment of the fixed claim, updates isolated one material relation, and source relations could be independently checked.Claims, passages, labels, and expected handling were frozen before model calls; provenance audits used publication identifiers, trial records, and study descriptions.Source: §4.1 · p. 4
  • The applicability and direction controls distinguish using a result for the wrong population from failing to register a relevant opposing result.CF-A3's olaparib evidence concerns first-line maintenance after platinum response, while CF-D3 contrasts fewer falls in initial evidence with more falls in the added high-dose vitamin-D trial.Source: §4.1 · p. 4
  • In CF-P2, a later QUEST analysis and a separate dupilumab trial both concern add-on treatment for uncontrolled asthma, making the choice a lineage decision.Different title and PMID notwithstanding, the later analysis shares trial NCT02414854; the independent trial is NCT01854047.Source: §4.1 · p. 5

4.2 Experimental unit and freezing

  • Matched conditions within a clinical family share one claim and one audited contrast, and repeated calls measure response stability within that family.The original and follow-up suites were frozen separately and reported as distinct stages.Source: §4.2 · p. 5

4.3 Evaluation suites and analysis status

  • The original suite contains 10 families and the separately frozen follow-up adds four, while post-hoc diagnostics and exploratory analyses are not pooled into the primary comparison.The original suite includes provenance, applicability, direction, and H3 families; follow-up adds pure-provenance and compositional families.Source: §4.3 · p. 5

5 Experiments and results

The experiments separate lineage recognition from evidence aggregation across clinical families, with repeated calls measuring within-input stability. GPT-5.5 recognizes independent updates yet can still over-credit same-trial reports; grouping repairs some conditions.

5.1 Experimental setup

  • Five model conditions cover repetition, lineage, applicability, direction, and order controls; structured outputs were validated, and rejected calls contribute no estimate for that outcome.Recognition candidates were presented in both orders, with five calls per order; grouped aggregation retained displayed documents and added study-level instructions.Source: §5.1 · p. 5

5.2 Models can recognize study lineage directly

  • Across CF-P1–P3, GPT-5.5 and Qwen identify the independent update in 10/10 recognition calls per family; Gemma and Llama are correct in 5/10.Ministral reaches 10/10 in CF-P1, while five of its CF-P2 and CF-P3 calls are rejected.Source: §5.2 · p. 5

5.3 Correct lineage recognition does not guarantee lineage-respecting aggregation

  • GPT-5.5 fully recognizes CF-P1–P3 yet over-credits same-trial updates in 5/5 aggregation calls per family; Qwen shows this contrast in CF-P1 and CF-P3.Qwen's CF-P2 aggregation outputs were rejected, so that comparison is unresolved.Source: §5.3 · p. 6
  • GPT-5.5 treats literal repetition and null controls as duplicate or unchanged in 45/45 calls, but marks same-trial publications as independent corroboration in 15/15 calls.The 15 calls are repeated measurements across three clinical families.Source: §5.3 · p. 6

5.4 Study grouping repairs some model–family conditions

  • In P5R, GPT-5.5 recognizes that START reports derive from the same trial in 10/10 calls, yet marks the follow-up independent in 4/5 document-level calls; study grouping reduces over-credit to 0/5.Qwen and Gemma remain at 5/5 over-credit after grouping.Source: §5.4 · p. 6

Correct lineage recognition does not guarantee lineage-respecting aggregation

In the later pure-provenance families, GPT-5.5's study-grouping condition removes same-trial over-credit while preserving independent-trial support. Other model outcomes vary, and compositional global flags do not isolate which study received credit.

Correct lineage recognition does not guarantee lineage-respecting aggregation

  • In P5R and P6R, GPT-5.5's same-trial over-credit falls from 4/5 document-level calls to 0/5 under study grouping.Figure 2 distinguishes original provenance families from later pure-provenance families and marks whether recognition and aggregation criteria were met.Source: Correct lineage recognition does not… · p. 7, Figure 2 · p. 7
  • In P6R, GPT-5.5’s over-credit falls from 4/5 document-level to 0/5 under grouping, and Ministral’s from 5/5 to 0/5; Qwen and Gemma remain at 5/5, while Llama’s comparison remains unresolved.Source: Correct lineage recognition does not… · p. 7
  • In P6R, all five models use the independent-trial update in 5/5 calls before and after grouping; GPT-5.5 shows the same pattern in P5R.Ministral's grouped independent-update outputs for P5R were rejected.Source: Correct lineage recognition does not… · p. 7
  • In H5 and H6, GPT-5.5's global corroboration flag changes, but existing independent evidence prevents that flag from identifying which study received credit.The study-level same-trial comparison is therefore reserved for P5R and P6R.Source: Correct lineage recognition does not… · p. 7

6 Further analysis

Specificity controls show distinct handling of out-of-scope and challenging evidence, while provenance cues repair some GPT-5.5 conditions. A secondary audit preserves the broad pattern under two prompt variants but does not establish general prompt robustness.

6.1 Specificity and compositional evidence

  • GPT-5.5 recognizes target updates in all six applicability and direction families (10/10 per family), ignores out-of-scope positives, and registers challenges as conflicting or mixed evidence (5/5 per family).In H3, GPT-5.5 and Qwen recognize all three relations across 30 calls, yet over-credit the same-trial WOSCOPS analysis in 4/5 and 5/5 valid aggregation calls, respectively.Source: §6.1 · pp. 7–8

6.2 Explicit provenance information repairs one model condition

  • For GPT-5.5 in CF-P3, explicit provenance or a general study-dependence policy removes the same-trial corroboration flag in all five valid calls while independent-trial support remains 5/5.Qwen retains the same-trial flag in all five calls under both oracle conditions; testing whether the model's own recognition improves later aggregation would require a separate two-stage design.Source: §6.2 · p. 8

6.3 Prompt-order and lineage-by-overlap controls

  • Under a reordered equivalent prompt, same-trial over-credit appears in 16/20 GPT-5.5 records and all 20 Qwen and Gemma records; this audit does not estimate general prompt robustness.In the exploratory ProMISe/ARISE control, GPT-5.5 handles all 20 prompted calls as expected, but selected overlap strata within one family do not identify a causal effect of textual overlap.Source: §6.3 · p. 8

7 Conclusion

The paper identifies a cross-task gap: correct lineage recognition need not guide a separate evidence-aggregation judgment. Its evidence is limited to audited families and displayed source metadata.

7 Conclusion

  • GPT-5.5 and selected Qwen3-4B and Ministral-3-8B conditions show that recognized study lineage can fail to guide separate evidence aggregation.In later pure-provenance families, grouping eliminates GPT-5.5 same-trial over-credit while retaining independent-evidence support; effects vary across other model conditions.Source: §7 · p. 8
  • The evaluation covers 14 source-audited clinical families, with repeated calls measuring within-input stability and staged suites separating the original, grouping follow-up, and exploratory diagnostics.The results support using underlying study identity as the unit of corroboration.Source: §7 · p. 8
  • The conclusions are scoped to audited families and displayed source metadata; broader prevalence and cue-free grouping require further evaluation.Source: §7 · p. 8

Limitations

The study’s scope is limited by selected clinical contrasts, repeated-call stability checks, and separate stateless tasks, which do not establish generality or reveal internal representations. Its frozen inputs and archived configurations support auditing, but exact reruns of GPT-5.5 require Codex access.

Limitations

  • The 14 selected clinical families are not representative of medical claims or publications; repeated calls assess within-input stability, and stateless tasks do not reveal internal representations.The grouping condition also exposes study identifiers and changes the prompt; rejected outputs limit some cross-model comparisons.Source: Limitations · p. 9

A.1 Ground truth, freezing, and experimental unit

  • The clinical family is the experimental unit because matched conditions share one claim and audited contrast; repeated calls measure response stability only.The original ten-family suite and later four-family package were frozen at different stages, and superseded and selected inputs were not pooled.Source: §A.1 · p. 10

A.2 Selected family inventory

  • Table 4 lists the selected 14-family comparison, while Table 5 gives one traceable publication anchor per initial state rather than every source or identity.A missing study identifier is not evidence that two reports describe the same study.Source: §A.2 · pp. 10–11

B Prompts, execution, and implementation details

  • Figures 3–5 show base prompt templates, but omit the varying clinical JSON payload and model-specific formatting suffix; recognition answers are not passed to verifier calls.The input placeholder is replaced with frozen claim, source-metadata, and evidence JSON, without gold answers or expected score movements.Source: B Prompts, execution, and… · p. 11

B.1 Inputs, output validation, and execution

  • Recognition and verifier calls use different inputs and response contracts, with schema, item identity, evidence coverage, order, and quotations validated before semantic scoring.Rejected outputs remain in denominator accounting but receive no semantic score.Source: §B.1 · p. 11

B.2 Model configuration and compute

  • Open-weight runs used greedy decoding, while GPT-5.5 ran through Codex without a specified temperature or random seed; model-specific adapters and settings differed.GPT-5.5’s parameter count and serving hardware, and reliable wall-clock and GPU-hour totals, are unavailable in archived records.Source: §B.1 · p. 11, §B.2 · p. 12

B.3 Call accounting

  • Repeated invocations of exact inputs measure response stability, not additional clinical families; secondary, diagnostic, and exploratory blocks remain outside the 14-family comparison.The original ten-family suite and later four-family package were executed in separate stages.Source: §B.3 · p. 12, §B.2 · p. 12

B.4 Artifact contents and rerun requirements

  • The artifact preserves source, inputs, prompts, schemas, configurations, audits, outputs, hashes, figures, and rerun scripts; exact GPT-5.5 reruns require Codex access.Open-weight conditions can be rerun from pinned model revisions and the recorded GPU runner.Source: §B.4 · p. 12

C Original provenance results by family

The original provenance results compare direct lineage recognition with separate same-trial aggregation, while later controls examine grouping, other relation types, and prompt-order robustness. These comparisons have distinct validity and interpretation boundaries.

C Original provenance results by family

  • Table 7 presents recognition and same-trial aggregation rows behind the central cross-task comparison; rejected calls receive no semantic score, so 0/0 is not a measured zero.The comparison concerns whether correct lineage answers coexist with appropriate handling in a separate aggregation task.Source: C Original provenance results by family · p. 12

D Study-grouping and later-family results

  • Table 8 separates same-trial over-credit from preservation of independent-trial support; H5/H6 global flags cannot isolate credit assigned specifically to a newly added same-trial report.Both H5/H6 baselines already contain an independent study.Source: D Study-grouping and later-family… · p. 16

E Applicability, direction, and compositional controls

  • The applicability and direction integration rows are GPT-5.5 medium results, and the open-weight package lacks a harmonized family-level summary for those axes, so no cross-model integration rate is inferred.The comparison-axis controls are distinct from the provenance recognition results.Source: E Applicability, direction, and… · p. 16
  • In held-out H3, GPT-5.5 and Qwen answer all 30 recognition calls correctly, yet flag the added same-trial WOSCOPS report as independent corroboration in 4/5 and 5/5 valid aggregation calls, respectively.The recognition calls comprise ten comparisons for each of provenance, applicability, and direction.Source: E Applicability, direction, and… · p. 16

F.1 Prompt-order robustness

  • Across two recorded prompt variants, GPT-5.5 marked same-trial updates independent in 16/20 records and Qwen in 20/20; Gemma’s same-trial records failed the provenance flag rule in 20/20.Gemma was canonically parseable in 40/40 records but strict-valid in 0/40; these calls were not pooled with primary denominators.Source: §F.1 · p. 17

F.2 Lineage-by-overlap construction and results

The lineage-by-overlap control uses source-faithful passages from one ProMISe/ARISE family and an explicitly stated study-level policy. GPT-5.5 followed the expected policy across overlap strata, but the design does not estimate a causal overlap effect or unprompted lineage discovery.

F.2 Lineage-by-overlap construction and results

  • The one-family control uses source-faithful ProMISe and ARISE passages, with a same-trial HTA report and an independent ARISE trial; inputs do not include source_group_id.The selected passages were not synthetically paraphrased, trimmed, or padded.Source: §F.2 · p. 17
  • Overlap was measured against the initial ProMISe numerical-result anchor using normalized word and number tokens with punctuation removed.The described tokenization forms the basis for the reported overlap measures.Source: §F.2 · p. 17
  • High- and low-overlap passages were defined using token-set Jaccard, ROUGE-L F1, and cosine similarity, with within-lineage separation thresholds; these strata are not randomized treatments.Estimating a causal overlap effect requires a separately designed intervention.Source: §F.2 · p. 17
  • GPT-5.5 produced the expected policy-defined handling in all four lineage-by-overlap cells, or 20/20 calls, with same-trial passages treated as duplicate evidence and independent-trial passages as corroboration.The verifier prompt explicitly states the study-level policy; the result tests policy application, not unprompted lineage discovery.Source: §F.2 · p. 17

G Post-hoc oracle and cue-ablation diagnostics

Post-hoc diagnostics test oracle cues and source-group metadata after the primary provenance failure, while validity tables distinguish semantic outcomes from rejected calls. These diagnostic results are not additional primary families.

G Post-hoc oracle and cue-ablation diagnostics

  • The CF-P3 oracle block supplies item-specific study relations or a policy against treating same-trial reports as independent corroboration; H3-M1 removes only source_group_id while retaining other evidence and metadata.Together, the two diagnostics contain 65 responses per tested model; Llama and Ministral were not run in this block.Source: G Post-hoc oracle and cue-ablation… · pp. 17–18
  • GPT-5.5 retains 10/10 provenance recognition in H3-M1 but flags the same-trial report in 5/5 verifier calls; in CF-P3, both oracle conditions remove that flag in 5/5 valid calls and preserve the independent-trial control.Qwen retains the flag under both CF-P3 oracles, while Gemma’s same-trial oracle outputs are rejected.Source: G Post-hoc oracle and cue-ablation… · p. 18

H Validity and rejection accounting

  • Table 16 reports post-hoc diagnostic outcomes rather than additional primary families; its 0/0 cells mean five rejected outputs and provide no semantic estimate.Qwen has five further rejected H3-M1 exact-repeat outputs, and Gemma has 25 rejected diagnostic outputs overall.Source: Table 16 · p. 20
  • Validity tables record response-contract validity rather than correctness, and rejected calls receive no semantic score; repeated calls remain within-input stability checks, not added empirical units.The 14 clinical families are the empirical units.Source: H Validity and rejection accounting · p. 18, Table 17 · p. 20

Related papers

Loading 2610.23721v1…