PaperlayerView exact revision on arXiv

Source-linked AI summary

Every Edge Needs a Quote: Building a Quote-Grounded Knowledge Graph of Healthcare Software Adoption with Language-Model Agents

Juan Felipe Duran, Arthur Tobler, Martin Vallières

arXiv:2610.32781v1cs.CLcs.AIcs.IR

TL;DR

Hospitals need evidence about which peers use healthcare software and what systems it connects to, but public evidence is scattered and extracted claims need verification. The paper builds a quote-grounded adoption and integration graph, reaching 0.94 precision and 0.71 recall against a page-level key, while 89% of relations it keeps beyond the golden reference are stated on a page.

  • Problem

    Evidence about peer use of healthcare software and connected systems is public but scattered, and incorrect customer listings can mislead buyers and misrepresent organisations.

    Source: §1 · p. 2

  • Method

    The pipeline builds adoption and integration edges from the web, requiring a verbatim page quote for each edge and using layered grounding to decide which quoted relations to keep.

    Source: §1 · p. 2, §3.3 · p. 5, §9 · p. 12

  • Results

    On 61 vendors, the pipeline reaches 0.94 precision and 0.71 recall against a page-level key, and 89% of relations it keeps beyond the golden reference are stated on a page.

    Source: §1 · p. 2

  • Takeaways & Limitations

    A confidence score built from recorded panel votes lets catalogue owners choose their preferred balance between precision and recall.

    Source: §1 · p. 2, §9 · p. 12

    The page-level key covers only extractor-proposed relations, and the evaluation uses one industry partner’s catalogue rather than other domains.

    Source: §8 · p. 11

Abstract

from arXiv · show

Hospitals regularly buy software that clinicians and patients depend on. Because these purchases are costly, hard to reverse and can affect patient care, buyers look for evidence of which peer institutions already use a product and which systems it connects to. This evidence is public but scattered across vendor websites, press releases and third-party pages. Language-model agents can now collect it at low cost, but the correctness of the extracted claims is difficult to establish, and a wrongly listed customer can mislead buyers and misrepresent an organisation. We present a pipeline that builds a knowledge graph of customer and integration relations from the open web under one rule: every edge must be supported by a verbatim quote from its source page. Pages come from a deterministic crawl of each vendor's site and a budgeted ReAct agent that searches the rest of the web. A permissive extractor proposes relations with quotes, and a layered grounding stage decides which to keep, using a referent step that reads the whole page when a quote names only one side, deterministic checks, and a panel of five model calls voting on nine criteria. On 61 vendors, the pipeline reaches 0.94 precision and 0.71 recall against a page-level key covering the 6,428 candidate relations proposed by its extractor, whose verdicts were drafted by one model and re-checked with a second. Against the golden reference, the marketplace's own curated list, it scores lower, but 89% of the relations it keeps that the reference lacks are stated on a page. A confidence score built from the panel's votes keeps more correct relations than the fixed rule at the same precision, and reaches 0.90 precision at 0.90 recall. We also discuss how narrow, bounded model calls, string-matched evidence and voting address long-context degradation and run-to-run variance.

1 Introduction

Hospitals need public evidence of which peers use healthcare software and what systems it connects to, but that evidence is scattered and model-extracted claims require verification. The paper presents a quote-grounded pipeline and evaluates it on two benchmarks.

1 Introduction

  • The pipeline addresses the risk of incorrect customer listings, which can mislead buyers and associate a hospital with a product it may not use.Evidence buyers seek includes peer institutions using a product and the systems it connects to.Source: §1 · p. 2
  • Every graph edge must have a verbatim quote from a retained page, with the page URL and retrieval time recorded.The quote makes the edge checkable but does not by itself establish that the relation is correct.Source: §1 · p. 2
  • 0.94 precision and 0.71 recall on the page-level key; 89% of relations absent from the golden reference were stated on a page.The page-level key covers 6,428 candidate relations across 61 vendors, with verdicts re-checked by a second model.Source: §1 · p. 2
  • A confidence score built from panel votes offers better operating points than the pipeline’s fixed decision rule.The score is presented as a way to choose among precision–recall trade-offs.Source: §1 · p. 2
  • The paper also offers design lessons on long-context degradation, run-to-run variance and prompt tuning.These lessons are discussed in Section 6.Source: §1 · p. 2

2 Problem Definition

The problem definition represents healthcare software adoption and integration as typed edges grounded in page text. Each edge records its quote, source page and retrieval time.

2 Problem Definition

  • The input is one target product and its vendor’s website; the output is a set of typed edges.For adoption, the subject is a health organisation and the object is the target product; for integration, the target product connects to another named product.Source: §2 · pp. 2–3
  • The edge’s relation type is Adopts or Integrates.Table 1 is identified as defining terminology used throughout the paper.Source: §2 · p. 3, Table 1 · p. 3
  • Each edge records a verbatim contiguous quote from the normalized page text, the page URL and retrieval time.The quote establishes what the page stated when retrieved, not that the relation is current or that the organisation endorses the product.Source: §2 · p. 3

3 Method

The pipeline discovers pages, extracts quoted customer and integration relations, and grounds candidates before writing surviving edges with quote and provenance. Its staged design combines deterministic checks with model judgments to decide whether a quoted relation is actually supported.

3 Method

  • The pipeline has three stages—discovery, extraction, and grounding—and writes surviving relations to a graph with their quote and source metadata.Rejections are also stored with reasons for later analysis.Source: §3 · p. 3, §3.4 · p. 5

3.1 Discovery

  • Vendor-site discovery uses a deterministic sweep, while a budgeted ReAct agent searches third-party pages and can use vendor-confirmed customers as search leads.The agent has hard search and page-read limits, and its pages pass through the same extraction and grounding as vendor pages.Source: Figure 1 · p. 3, §3.1 · pp. 3–4

3.2 Extraction

  • For each page and relation type, a permissive structured-output extractor proposes relations with contiguous quotes and enough adjacent text to show names and the relation cue.Its instructions distinguish adoption from investment, awards, or network membership.Source: §3.2 · pp. 4–5

3.3 Grounding

  • When a quote matches the page but is non-contiguous, splice repair replaces it with the page’s span; the referent then reads the whole page when a quote omits an endpoint or was repaired.The referent returns an argument, verdict, and proof sentence copied from the page.Source: §3.3 · p. 5
  • Fourteen deterministic shape checks reject structural failures, then five model calls vote on applicable semantic criteria using a quote-matched section capped at 6,000 characters.A criterion passes with at least half of the returned votes marked yes.Source: §3.3 · p. 5

3.4 Persistence

  • Surviving relations become graph edges carrying the quote, source URL, retrieval time, extraction identifier, and prompt version; each rejection is stored with its reason.Source: §3.4 · p. 5

4 Experimental Setup

The evaluation uses a page-level key and the marketplace’s curated list as benchmarks with different coverage. It describes the corpus, how candidate relations were judged, and what each benchmark can measure.

Corpus

  • 6,428 candidate relations were judged from 3,284 pages covering 61 products, after one candidate was excluded as unjudgeable.The final results include candidates affected by two grounding components that were finalized after the initial pass and re-run.Source: §4 · p. 6

Golden reference

  • The marketplace reference contains 749 customers and 249 integrations, but its omissions make recall a lower bound and precision too low.Its unit is the vendor-organisation pair, with page-level output collapsed before scoring.Source: §4 · p. 6

Page-level key

  • The page-level key labels each proposed page-relation pair, and the second model re-checked every verdict, reversing 911 of 6,428 judgments (14%).The first author read a subset, but the records do not identify which entries.Source: §4 · p. 7

Metrics

  • The page-level key measures grounding only over extractor proposals, whereas the golden reference also reflects extraction recall but mixes it with reference gaps.Source: §4 · p. 7

5 Results

The results compare the pipeline with a page-level key and the marketplace’s curated reference, then examine rejection patterns, the referent’s contribution, and confidence-score trade-offs.

5.1 Overall Accuracy

  • 94% precision and 0.71 recall on the page-level key measure accuracy over extractor-proposed candidates.The same output scores 0.34 precision and 0.41 recall against the golden reference; 89% of kept pairs missing from that reference are stated on a page.Source: §5.1 · p. 7, Table 3 · p. 7
  • 37% of customers whose only proof was on third-party pages were found by the discovery agent.It found pages proving 88 of 239 such customers.Source: §5.1 · p. 8

5.2 Error Analysis

  • 1,106 of the 1,398 correct relations missed were rejected by the semantic panel, versus 276 by the referent.The panel’s 6,000-character context limit can miss evidence elsewhere on a page, including headings above logos or testimonial attributions.Source: §5.2 · p. 8, Table 4 · p. 8

5.3 Effect of the Referent

  • Recall rose from 0.361 to 0.808 when the referent was added on 266 vendor pages, while precision rose from 0.887 to 0.932.The page-level key detects this gain; the golden-reference F1 barely changed because most recovered relations had no golden row.Source: §5.3 · p. 8, Table 5 · p. 8

5.4 Confidence Score

  • At 0.90 precision, the confidence score reaches 0.90 recall; at 0.94 precision, it reaches about 0.77 recall versus 0.63 for the fixed rule.The score is fitted offline from recorded signals and evaluated out of sample; owners can set the threshold and send lower-scoring relations for human review.Source: §5.4 · pp. 8–9, Figure 3 · p. 9, Table 6 · p. 9
  • Most confidence-score signal comes from panel votes, while no signal describes the evidence page type where remaining errors concentrate.The customer-relation flag carries the heaviest weight in most fitted models, reflecting this corpus’s base rate rather than evidence about the relation.Source: §5.4 · pp. 9–10

6 Discussion

Longer model inputs and outputs degrade performance, while repeated calls can vary sharply. The pipeline responds with bounded, task-specific calls, deterministic checks, per-criterion voting, and a controlled prompt-tuning protocol.

6.1 Long-Context Degradation

  • The extractor’s splicing rate rises from 6 in 100 quotes for one relation to 15 in 100 for four to seven, motivating bounded calls and separating reading from judging.The referent accepted 162 of 176 true relations without the extractor’s quote, versus 141 with it, suggesting a weak quote can anchor the model.Source: §6.1 · p. 10
  • Most correct relations rejected by the panel after referent confirmation have evidence outside its excerpt, exposing a cost of bounded inputs.Widening the panel’s view or treating referent confirmation as evidence is identified as the most direct fix.Source: §6.1 · p. 10

6.2 Run-to-Run Variance

  • Three identical extraction calls on one page returned 1, 39, and 2 relations; five calls per relation vote on each criterion, and vote shares are the confidence score’s strongest features.The pipeline records every vote, and prompt comparisons use several runs on frozen inputs.Source: §6.2 · p. 10

6.3 Prompt Tuning

  • The prompt-tuning protocol freezes cases, labels and reviews them, repeats paired comparisons up to seven runs per finalist, then confirms the chosen version end to end.Prompts are versioned, and changing a grounding prompt triggers extraction under a new version.Source: §6.3 · p. 10
  • For 216 vendor-page customer relations, separate vendor-page prompts reached F1 0.975 over seven runs, versus 0.955 for a generic prompt over three (Wilcoxon signed-rank p=0.014).Other prompt principles include separate yes/no criteria, explicit exclusions, worked referent examples, and verbatim proof sentences for string matching.Source: §6.3 · p. 11

7 Related Work

The paper situates its graph construction among knowledge-base extraction, attribution and verification, and agent-based judging. It adopts attribution at edge creation and uses a criterion-voting panel to filter relations.

Knowledge-graph construction

  • Knowledge Vault fuses noisy extractions into probabilistic facts, while distant supervision inherits knowledge-base gaps as false negatives in a positive–unlabelled setting.The paper places its use case in language-model relation extraction and graph construction [1] [20] [25], and notes GraphJudge’s model-based filtering of candidate triples.Source: §7 · p. 11

Attribution and verification

  • Prior attribution work formalises when a statement is supported by an identified source; this pipeline applies attribution when each graph edge is written and verifies against the unmodified page.Related approaches ask models to cite [7], revise outputs against retrieved evidence [6], or score factual precision of atomic claims [14]; Ji et al. [10] survey unsupported generation.Source: §7 · p. 11

Agents and model judges

  • The discovery agent follows ReAct [27], while WebGPT is a browsing-agent example [16] and retrieval-augmented generation is the broader family [12].Because model judges can be biased and inconsistent [28], the paper uses per-criterion panel voting and records every vote; majority voting over samples can improve reliability [26].Source: §7 · p. 11

Quality of evaluation data

  • The paper’s page-level key is re-checked, while its confidence score is cross-validated; calibration in Guo et al.’s sense [8] remains a next step.This relates to concerns that test-label errors can change which model appears best [18], and to auditing evaluation data [21] and system-level analysis.Source: §7 · p. 11

8 Limitations and Responsible Use

The evaluation is limited by candidate coverage, label-review uncertainty, a narrow vendor domain, and measurement choices. A quote establishes what a page stated at retrieval, not current use or endorsement, so records need human review before purchasing decisions.

Benchmarks

  • The page-level key covers only extractor-proposed relations, and its verdicts were re-checked by a second model but only partly by a person, without measured agreement.Reviewers saw the judge’s verdicts, which may anchor agreement, and the pipeline and judge models come from the same provider; a human-annotated sample with measured agreement could bound key error.Source: §8 · p. 11
  • The golden reference comes from the industry partner’s catalogue, all evaluated vendors are drawn from it, and other domains were not evaluated.The golden reference is maintained by the partner for whom the system was built.Source: §8 · p. 11

Measurement

  • Final scores combine the corpus pass with a rerun of candidates affected by final repairs, while the confidence score was fitted on one corpus and some measurements use only one to seven runs.The fitted score uses signals from the corpus pass rather than a fresh pass.Source: §8 · p. 11

What a quote establishes

  • A quote establishes that a page stated a relation when retrieved, not that it remains current or that the named organisation endorses the product; the graph does not yet model time beyond retrieval dates.Image-rendered logo walls are only partly readable through alternative text, and incomplete entity resolution can leave one hospital under multiple names.Source: §8 · p. 11
  • Records should be reviewed by a person before informing a purchase, and no threshold for publishing without review has been validated.Source: §8 · p. 12

9 Conclusion

The conclusion presents a quote-grounded adoption and integration graph, with bounded model calls and deterministic checks deciding which relations to retain. On 61 vendors it reports 0.94 precision and 0.71 recall against a re-checked page-level key, and a vote-based confidence score lets catalogue owners choose a precision–recall balance.

9 Conclusion

  • On 61 vendors, the pipeline reaches 0.94 precision and 0.71 recall against a re-checked page-level key.Source: §9 · p. 12
  • The pipeline builds a healthcare software adoption and integration graph in which every edge points to its source sentence and page.Narrow model calls and deterministic checks help decide which quoted relations to keep.Source: §9 · p. 12
  • Most relations found beyond the golden reference are stated on a page, and a confidence score built from recorded votes lets catalogue owners choose their precision–recall balance.Source: §9 · p. 12

Code Availability

The repository makes the pipeline and a small sample available, but a rerun will not reproduce the reported full-corpus results.

Code Availability

  • The repository includes the full pipeline, public adaptations of its 20 prompts, ten illustrative products, and public editions of the experiment write-ups.The sample contains 139 customer and 21 integration labels.Source: Code Availability · p. 12
  • The test suite runs without API keys or network access, and the ten-product sample can be run with users’ own keys for fetching pages and making model calls.Source: Code Availability · p. 12
  • Because prompts are adapted, the sample is small, and web pages change, rerunning will not reproduce the reported full-corpus results.The artefacts behind those results are listed in Appendix E.Source: Code Availability · p. 12

Use of AI Assistants

The appendix details a versioned, multi-stage pipeline and its grounding prompts, then describes confidence scoring from recorded signals. Claude Opus 5 re-checked the page-level key and assisted with code and manuscript drafting.

A Implementation Details

  • The outer workflow crawls vendor sites, runs extraction, then starts relation-specific discovery agents; the inner workflow resolves endpoints, grounds candidates, and writes surviving edges.Pages are cached, URLs are normalized and filtered, and discovery selections are checked against observed URLs; Claude Opus 5 was used to re-check the page-level key, help write code, and help draft the manuscript.Source: Use of AI Assistants · p. 12, A Implementation Details · p. 15

B Grounding Checks and Criteria

C Prompts

C.1 Judge Prompt for Customer Relations

  • The customer judge counts actual use or provision, while excluding relationships such as investment, planned use, or a page naming only an individual.It returns a YES or NO with a sentence of justification.Source: §C.1 · p. 17

C.2 Judge Prompt for Integration Relations

  • The integration judge requires a stated technical connection between two named products, not generic compatibility, commercial ties, or mere co-appearance.Planned connections and general-purpose infrastructure connections are also excluded.Source: §C.2 · p. 17

C.3 Referent Prompt for Customer Relations on Vendor Pages (Abridged)

  • For vendor-page customer relations, the referent prompt checks whether the page supports adoption, distinguishing vendor-presented case studies from bare logos and third-party listings.It requests a reason, a yes/no verdict, and a verbatim proof sentence; the test asks whether someone at the organisation would recognize the page as describing its use.Source: §C.3 · pp. 17–18

D Confidence Score Details

  • The confidence score uses 22 recorded signals, and the search compares regularised logistic regression and shallow decision trees using out-of-sample five-fold cross-validation.The search evaluated 603,500 subsets of up to eight features, extended the forty best by beam search, and separately searched 141,724 trees; it took about two hours and no model calls.Source: D Confidence Score Details · p. 18
  • 0.907 is the best F1, reached with thirteen features; these model-selection results are optimistic because candidates were chosen and scored on the same folds.The best five-feature model reaches 0.898 and the best tree 0.900; unanimous-panel relations are 0.89–0.97 correct, while split-panel relations are 0.89 correct when the referent confirms and 0.33 when it refuses.Source: D Confidence Score Details · p. 19

E Reproducibility

The full-corpus artefacts are not all publicly available, which limits independent access to the materials underlying the reported quantities.

E Reproducibility

  • The public repository does not contain the full-corpus data; the pipeline, scoring scripts, and write-ups used for the paper are held in the industry partner’s private repository.The corpus table is excluded from that repository because of its size.Source: E Reproducibility · p. 19
  • The corpus table contains 6,429 candidates with votes, referent decisions, rejection reasons, run metadata, and outcomes of rerun candidates.Source: E Reproducibility · p. 19
  • The private artefacts also include 6,428 judge verdicts and their review, benchmark scoring and confidence-score scripts, and a golden-reference snapshot covering 61 products.The snapshot lists 749 customer and 249 integration rows.Source: E Reproducibility · p. 19

Related papers

Loading 2610.32781v1…