PaperlayerView exact revision on arXiv

Source-linked AI summary

EviStreams: Human-in-the-Loop AI Data Extraction for Systematic Reviews in Medicine

Sai Karthik Kosuri, Ankita Shashikant Bhosale, Michael Glick, Alonso Carrasco-Labra, Chris Callison-Burch

arXiv:2609.27418v1cs.CL

TL;DR

Systematic-review extraction is labor-intensive and must preserve independent review and an auditable record. EviStreams places human review at program design, field specification, and extraction adjudication; across four clinical corpora, field specification mattered more to extraction quality than model choice.

  • Problem

    Systematic-review data extraction is labor-intensive, and AI assistance must fit established protocols for independent review, adjudication, blinding, and auditability.

    Source: §1 · p. 1

  • Method

    EviStreams is a no-code platform where experts approve extraction plans, calibrate field specifications, and use reviewer-blinded dual review with adjudication.

    Source: §1 · pp. 1–2, §2.4 · p. 3

  • Results

    Field specification mattered more than model choice; richer specifications improved macro-F1 in 40 of 51 model–form comparisons (78%).

    Source: §1 · p. 1, §3.2 · p. 5

  • Takeaways & Limitations

    Expert effort is best spent on field specification, with adjudication the second-most-useful place for human intervention to catch residual errors.

    Source: §6 · p. 7

    The evaluation covers four review areas; the authors are extending it to test whether field specification remains the biggest lever elsewhere.

    Source: Limitations · p. 7

Abstract

from arXiv · show

Systematic reviews underpin clinical guidelines, yet their data-extraction step is a major expert-labor bottleneck bound by a protocolized workflow: two reviewers extract each study independently, an adjudicator resolves disagreements, and the team keeps an auditable record of how every value was produced. Large language models can assist with extraction, but that assistance must fit established review protocols and preserve reproducibility. We present EviStreams, a live, open-source, no-code web platform that puts review teams in control of AI-assisted extraction at three key stages: program design (a structured decomposition approved before any code runs), field specification (typed field definitions calibrated from a pilot), and extracted predictions (reviewer-blinded dual review with adjudication). Working through a form builder, a domain expert defines typed fields rather than prompts, runs extraction over uploaded PDFs, inspects every value alongside the supporting passage it came from, and resolves a reviewer-blinded dual review into an auditable consensus export. An evaluation across four clinical corpora and three frontier model families, released with the system, shows that extraction quality is shaped far more by the field specification than by the choice of model. EviStreams is live at https://evistreams.com/demo and released under Apache-2.0.

1 Introduction

EviStreams addresses the labor-intensive, protocol-bound extraction step with a human-controlled AI workflow that preserves source traceability. Its evaluation finds field specification matters more for extraction quality than model choice.

1 Introduction

  • Systematic-review extraction is labor-intensive and must fit a protocol of independent review, adjudication, blinding, and auditable records.Extraction errors can propagate into pooled estimates and clinical recommendations.Source: §1 · p. 1
  • Extraction quality is governed more by task specification than by model choice across the four clinical corpora studied.Model or pipeline changes shifted performance by only a few F1 points, with mixed direction across corpora; richer specifications produced better results.Source: §1 · p. 1
  • EviStreams is a live, open-source, no-code platform that puts domain experts in control of AI-assisted extraction.Reviewers approve or revise a proposed extraction plan before code runs, refine field definitions from pilot results, and conduct blinded dual review with adjudication.Source: §1 · pp. 1–2
  • Every AI-extracted value is paired with its verbatim source quote, and the system supports control from extraction planning through final adjudication.Reviewers can inspect the quote associated with each value.Source: §1 · p. 2

2 The EVISTREAMS System

EviStreams supports systematic reviews with a methodologist-approved extraction plan, editable structured field specifications, and reviewer-blinded dual review with adjudication.

2 The EVISTREAMS System

  • Designed for the ADA Living Guidelines Program, EviStreams supports methodologist-designed forms, PDF extraction, and adjudication end to end.A methodologist approves the proposed pipeline, then reviewers run extraction over uploaded PDFs and adjudicate results.Source: §2 · p. 2

2.1 Extraction pipeline

  • The system groups related fields into extraction stages, then compiles an approved plan into predefined extraction code without LLM-written code.Independent groups run in parallel, and later stages can use earlier results.Source: §2.1 · p. 2, Figure 2 · p. 2

2.2 Structured field specification

  • Each field separates its user-owned concept from evidence-finding hints, hard output rules, and examples that anchor the expected value shape.Separating meaning from retrieval helps methodologists localize errors to one element; hints and rules can be automatically generated and refined after pilots.Source: §2.2 · pp. 2–3

2.3 Table extraction

  • For medical-study tables, EviStreams extracts anchor columns that identify each row, then fills remaining cells with a separate call per row.This approach is designed to reduce row and column drift associated with naive table extraction.Source: §2.3 · p. 3

2.4 Human review at three key stages

  • The proposed extraction plan is mechanically checked for cycles and missing or duplicated field assignments, then held for reviewer approval.After testing the form on typically three to five papers, reviewers can revise field descriptions, hints, or rules and update prompts at runtime.Source: §2.4 · p. 3
  • Two blinded reviewers independently complete the same form, and an adjudicator resolves field-level disagreements while the system preserves the value history and source quotes.Reviewers see the AI pre-fill but not each other’s entries; when changing an AI value, they attach their own supporting quote.Source: §2.4 · pp. 3–4, Figure 4 · p. 3

Implementation

  • The implementation uses a Next.js and FastAPI stack with Celery workers, Redis-backed WebSocket progress, and Supabase storage.The system also uses Markdown ingestion and a multi-model fallback chain over three frontier families.Source: §2.4 · p. 4

3 Evaluation

Across four clinical corpora, extraction quality varied more with the field specification than with model choice. The evaluation also found that many reviewed discrepancies reflected methodology rather than model errors.

3 Evaluation

  • The evaluation used four clinical corpora with protocol-derived references, type-matched field scoring, and row alignment; ground truth could contain reviewer conventions or errors.The free-text judge was one of the three extraction models, a potential circularity; it decided 34% of scored comparisons.Source: §3 · p. 4

3.1 Production quality and robustness

  • Claude and Gemini were within 0.03 micro-F1 on every corpus; GPT trailed by up to ∼0.05 on antibiotic and periodontitis.The between-corpus range (0.741–0.898) dwarfed the between-model range; Table 1 reports overall micro-F1 per corpus.Source: §3.1 · p. 4, Table 1 · p. 5
  • Collapsing the multi-signature pipeline into one hand-tuned prompt changed overall F1 by at most 0.036 in mixed directions, so decomposition showed no consistent accuracy gains.The sizable antibiotic-table swing was GPT-specific: ∆= −0.096 for GPT versus −0.005 for Claude and Gemini.Source: §3.1 · p. 5

3.2 Field specification is the largest observed lever

  • Richer field specifications improved macro-F1 in 40 of 51 model–form comparisons (78%), with mean ∆F1 0.049 and median 0.014.Excluding the oral-cancer index_test form, the mean improvement was 0.023 and the full specification remained better in 37 of 48 comparisons (77%).Source: §3.2 · p. 5

3.3 Error profile

  • Two-thirds of 1,003 reviewed discrepancies were methodological; fewer than a third were genuine model errors, concentrated in dense multi-arm, multi-table reporting.Table 2 percentages are shares of reviewed discrepancies, not extraction error rates.Source: §3.3 · p. 5, Table 2 · p. 5

4 Demonstration

The demonstration walks a methodologist through uploading PDFs, defining a form, approving an extraction plan, and calibrating fields on pilot papers. Reviewers then check AI pre-fills independently before an adjudicator resolves conflicts into a consensus export.

4 Demonstration

  • After the methodologist names fields and provides descriptions or examples, EVISTREAMS proposes a plan grouping related fields into parallel extraction stages for review before code runs.The reviewer can approve, edit, or reject the plan.Source: §4 · p. 5
  • A pilot over 3–5 papers lets the methodologist compare AI values with expected values and refine hints, rules, or examples before the next run.Edits save live and affect the next pilot run.Source: §4 · p. 5
  • Two blinded reviewers independently complete the same form on each paper, accepting or correcting AI pre-fills; an adjudicator resolves field-level conflicts into a consensus export.The export records the AI suggestion, both reviewer entries, the adjudicated value, and inter-rater agreement.Source: §4 · p. 6
  • The platform targets systematic-review clinical experts and methodologists, with NLP researchers as a secondary audience for the per-field decision chain.That chain comprises AI pre-fill, reviewer entries, and adjudication.Source: §4 · p. 6

5 Related Work

Related systems offer overlapping extraction assistance, but EVISTREAMS combines program-level review, specification calibration, and blinded dual review in one workflow. Its related-work discussion distinguishes that integration from novelty in individual pipeline techniques.

5 Related Work

  • SCHEMATIQ supports expert revision of a schema and extracted table, while AUTOFOREST targets Cochrane forest-plot generation from PDFs.SCHEMATIQ is described as EVISTREAMS’ closest general-purpose sibling.Source: §5 · p. 6
  • Established review and annotation tools let users correct predictions, whereas EVISTREAMS also lets reviewers inspect and revise the program before extraction runs.The cited earlier tools place the human loop at prediction level only.Source: §5 · p. 6
  • Table 3 compares extraction-assistance capabilities across systems, while EVISTREAMS is distinguished by program-level review combined with specification calibration and blinded dual review.The comparison is of capabilities because systems target different endpoints and corpora and no shared extraction benchmark exists.Source: §5 · p. 6, Table 3 · p. 6
  • The related-work discussion positions the central thesis as objective articulation mattering more than model choice, citing Hein et al. (2025) and Li et al. (2026).Concurrent clinical information-extraction work also explored modular or agent-based decomposition (Aal Abdulsalam et al., 2026; Hart and Bergamaschi, 2026).Source: §5 · p. 6
  • EVISTREAMS exposes user-defined structured fields in a live loop, unlike VAXSCOPE’s fixed taxonomy; the paper does not claim novelty in decomposition or source grounding alone.Its stated contribution is unifying human review over an editable specification and showing field specification to be the largest, most consistent quality control surface across its corpora.Source: §5 · pp. 6–7

6 Conclusion

Across four clinical corpora and three frontier model families, EVISTREAMS finds that structured field specification matters more to extraction quality than model or pipeline decomposition. It pairs that result with an expert-controlled, auditable workflow for systematic reviews.

6 Conclusion

  • Across four clinical corpora and three frontier model families, extraction quality is robust to model and pipeline decomposition, while structured field specification drives the biggest variance.The evaluation identifies specification as the most valuable human-effort lever.Source: §6 · p. 7
  • Expert effort is best spent refining field specifications, with adjudication the second-most-useful intervention for catching residual errors.Specifications can be edited immediately without regenerating the pipeline.Source: §6 · p. 7
  • EVISTREAMS combines a user-approved extraction program, iteratively refined specifications, and reviewer-blinded dual review to produce auditable systematic-review extractions.The workflow is intended to reduce transcription work and support systematic-review workflows.Source: §6 · p. 7

Limitations

The evaluation has limitations in reference-data agreement, specification calibration, evaluator independence, and corpus scope. In particular, performance without review-specific pilot calibration and generalization to further review areas remain untested.

Limitations

  • Inter-annotator agreement was not measured when the reference data was created.The available passages therefore do not report a human-agreement measure for that reference data.Source: Limitations · p. 7
  • Specifications were calibrated using pilot papers from the same reviews evaluated, so performance on a new review without prior calibration remains untested.The reported results measure review-specific, calibrated specifications.Source: Limitations · p. 7
  • The free-text judge shares a provider family with an extraction model, creating potential evaluator dependence that the authors do not quantify.Source: Limitations · p. 7
  • The evaluation covers 125 studies from four completed, peer-reviewed reviews; testing in further review areas is ongoing to assess whether specification remains the biggest lever.Source: Limitations · p. 7

Ethics Statement

EVISTREAMS structurally requires human verification before AI-extracted values enter a final dataset and records the path from suggestion to approved value. The paper also describes data handling and the model coverage used in the ablation evaluation.

Ethics Statement

  • No AI-extracted value enters a final dataset without human review, because dual review and adjudication enforce verification structurally.The per-field decision chain records each step from AI suggestion to human-approved value.Source: Ethics Statement · p. 8
  • The evaluation uses published peer-reviewed articles rather than patient-level or identifiable data, but uploaded PDF text is sent to commercial LLM APIs under provider data-handling terms.The hosted instance limits project data to invited team members through role-based access control.Source: Ethics Statement · p. 8
  • The authors distinguish genuine model errors from ground-truth and annotation-convention discrepancies because unchecked extraction errors could propagate into clinical guidance.The error analysis is reported openly, and the decision chain records the AI suggestion through human approval.Source: Ethics Statement · p. 8

A Ablation and Model Coverage

  • Claude, GPT, and Gemini were each tested on all four corpora using the production pipeline, a single-prompt decomposition ablation, and a description-only specification ablation.Source: A Ablation and Model Coverage · p. 11

B Per-Corpus Production Results

Table 4 reports per-form macro-F1 for three model families across four corpora under a shared pipeline and identical form specifications. It uses a canonical 29-field set for periodontitis and excludes optional and structural row-matching fields.

B Per-Corpus Production Results

  • The evaluation's record-level interventions and outcomes represent one row per arm, or one row per arm by time point.Source: B Per-Corpus Production Results · p. 11
  • Table 4 compares per-form macro-F1 for three model families across four corpora, using the same pipeline and identical form specifications.The caption indicates that best per-row results are bolded, with ties bolded for both models.Source: Table 4 · p. 11
  • Periodontitis scores use the canonical 29-field set, excluding optional fields and structural row-matching fields.Source: Table 4 · p. 11

C Specification Ablation, Full Detail

The specification ablation reports per-form macro-F1 differences between the full and description-only specifications. The oral-cancer index_test recall collapse is reproduced across all three model families.

C Specification Ablation, Full Detail

D Decomposition Ablation

The decomposition ablation breaks down results by form modality, while the detailed error analysis expands the discrepancy taxonomy with definitions and causes.

D Decomposition Ablation

  • Table 6 breaks down overall full-pipeline versus single-prompt ∆F1 between table and scalar forms, averaged across Claude, GPT, and Gemini.Positive values favor the full pipeline; the passage identifies this as the breakdown behind the robustness finding.Source: Table 6 · p. 11, D Decomposition Ablation · p. 11

E Detailed Error Analysis

  • The pipeline figure marks human review at program design, specification, and prediction stages.These are labeled HITL #1, #2, and #3, respectively.Source: Figure 5 · p. 12
  • Table 7 expands the discrepancy taxonomy with definitions and causes for each type.The table gives the definitions and causes; counts and shares appear in Table 2.Source: E Detailed Error Analysis · p. 11, Table 7 · p. 12
Loading 2609.27418v1…