AI-assisted Protocol Information Extraction For Improved Accuracy and Efficiency in Clinical Trial Workflows

1 Introduction

• Complexity, variable document quality, and frequent amendments make protocol extraction time-intensive and inconsistent despite its potential downstream benefits.Structured extraction may improve efficiency, documentation quality, ethical review, and compliance.Source: §1 · p. 2
• Protocol structuring remains manual and fragmented, creating time costs and avoidable variability, incompleteness, and inaccuracies.Specialized tools address only parts of the workflow, while general-purpose ad-hoc solutions typically lack integration, repeatability, performance, and compliance.Source: §1 · p. 2
• LLMs may assist established workflows by generating standardized first drafts for expert verification, freeing teams for adjudication and oversight.The passage also identifies consistency and auditability of protocol-derived data as potential benefits.Source: §1 · p. 2
• The paper presents a clinical-trial-specific RAG system with a Schedule of Events methodology for extensive automated protocol information extraction.The approach combines text-based domain-specific RAG with vision-based methods for tabular Schedule of Events data.Source: §1 · p. 3
• The evaluation compares RAG with standalone LLMs on 23 protocols and measures accuracy and operational impact with 13 CRCs.It also uses an LLM-assisted annotation and adjudication framework to assess hundreds of semi-structured data fields.Source: §1 · p. 3

2 Related work

• The paper characterizes its approach as broader than prior extraction of eligibility criteria and Schedule of Events information.Earlier work addressed selected features and SoE extraction, while one example extracted semi-structured oncology-record information.Source: §2 · p. 3
• Standalone LLM extraction can be constrained by document length, missed details scattered across sections, and tradeoffs between prompt count, cost, and completion time.The passage also identifies a lack of element-specific context and referencing as a challenge.Source: §2 · p. 4
• RAG combines LLM knowledge with element-specific retrieval queries, context, and generation prompts to address standalone extraction limitations.The paper notes that specialized context, terminology, output requirements, and section references may matter for protocol extraction.Source: §2 · pp. 4–5
• Schedule of Events tables pose extraction challenges because multi-page spans, merged cells, and hierarchical visits rely on visual layout.The paper describes a two-stage response: detect tables on protocol pages, then use multimodal vision-based generation to extract their information.Source: §2 · p. 5
• LLM-based evaluation has been used in prior work, and this study applies an LLM-based framework to assess abstraction accuracy and completeness.The paper presents expert evaluation as resource-intensive while quality standards remain important.Source: §2 · p. 5
• Expert annotation may be subjective and incomplete, and a small expert group cannot guarantee exhaustive ground truth.Adding human reviewers could improve completeness but is described as prohibitively labor-intensive and unscalable; the study uses LLM adjudication with human review of low-confidence cases and sampled quality control.Source: §2 · pp. 5–6

3.1 Protocol documents selection

• The study sample comprises randomly selected interventional drug studies with treatment as the primary purpose from Canada or the United States, while the controlled experiment assigns extraction tasks to 13 CRCs.The sample includes 9 oncology, 7 cardiovascular, and 7 other therapeutic-area studies; the CRCs had varying experience levels and received instructions and output examples.Source: §3.1 · p. 6, §3.6.1 · p. 15, §3.6 · p. 15

3.2 Extracted semi-structured data models

• The extraction models organize protocol information as JSON-representable data across six categories, including study details, interventions, schedules, eligibility, adverse events, and site requirements.The defined fields include objectives and endpoints, treatment details, visit procedures, safety reporting and monitoring, and site equipment and product storage.Source: §3.2 · p. 6, §3.2.1 · p. 7, §3.2.2 · p. 7

3.3.1 RAG extraction

• The RAG pipeline uses element-specific retrieval and prompts to generate structured outputs from relevant protocol chunks, supporting long documents, traceability, and human review.Context-aware chunking, reunion of split sections, and document ordering preserve structure and relationships between retrieved information.Source: §3.3.1 · pp. 8, 10

3.3.3 Schedule of Events table detection and extraction

• SoE extraction uses table detection followed by multimodal LLM extraction to preserve visual relationships among visits, procedures, and timepoints.The Microsoft Table Transformer identifies tables, whose page images are passed to a multimodal LLM with SoE-specific instructions; the controlled experiment uses the earlier gpt-4o configuration.Source: §3.3.3 · p. 11

3.4 Evaluation

• The evaluation compares protocol-derived JSON outputs with expert-supported ground truth using independent, granular LLM-as-judge scoring and predefined weighted category scores.Scoring emphasizes completeness and precision rather than style, tolerates non-contradictory extra detail, and weights higher-priority elements more heavily.Source: §3.4 · pp. 11–12
• The LLM judge agreed strongly with human scores across 95 JSON pairs, with Pearson r = 0.923, Spearman ρ = 0.942, linear κ = 0.699, and quadratic κ = 0.877.Its mean bias was −0.47 points on a 0–5 scale, indicating slightly lower scores than the human evaluator.Source: §3.4 · p. 12, Table 2 · p. 13

3.5 Data annotation

• Across 23 protocols, a human-annotated full dataset covering all data elements and categories is paired with gpt-4o and Claude 3 candidate datasets, then gpt-5 adjudicates using protocol text and extraction instructions.The final JSON output may be either submitted candidate annotation or a new annotation synthesized by the adjudicator; it includes a confidence score from 0–100.Source: §3.5.1 · p. 14, §3.5.2 · p. 14

3.5.3 Human-in-the-loop validation

• Human reviewers confirmed 87% (N = 35) of the 40 lowest-confidence adjudications and 100% (N = 24) of the randomly sampled cases.The authors estimate that this hybrid process yields ground-truth accuracy comparable to fully manual double-blind annotation.Source: §3.5.3 · pp. 14–15

3.6.1 Experiment design

• Each CRC completed two tasks using randomly assigned protocols and four randomly selected data-element categories, entering each value into a spreadsheet cell.Protocols were chosen from 23 available, and categories from six described in Section 3.2.Source: §3.6.1 · p. 15
• One task used RAG-prefilled values for CRC review, completion, and correction; the other required manual extraction, with task order randomized.The prefilled data used the experiment-era gpt-4o-mini configuration, not the later Claude Sonnet 4.5-based system evaluated in Section 4.1.Source: §3.6.1 · pp. 15–16
• After each task, CRCs reported completion time and ratings of mental demand and effort; after both, they answered five questions on satisfaction and experience with AI assistance.The satisfaction and experience questions are referenced in Table 8.Source: §3.6.1 · p. 16

4.1 Specialized RAG vs. standalone LLM performance

• Later RAG configurations achieved 89.0% average accuracy, versus 62.6% for standalone LLMs with fine-tuned prompting; the gpt-4o RAG process averaged 69.6%.Source: §4.1 · p. 16
• RAG gains were strongest for adverse events, site information, and interventions; general and inclusion/exclusion performance was relatively close, with chunk size affecting eligibility extraction.RAG 2’s inclusion/exclusion score was 77.6% versus 91.7% for RAG 1, partly attributed to the smaller chunk-size limit.Source: §4.1 · p. 17

4.1.1 Observations and examples

• Targeted retrieval maintained performance in long protocols and retrieved the correct adverse-event definition where standalone extraction focused on bleeding classifications.Source: §4.1.1 · pp. 18–19

4.2 Controlled experiment results, AI-assisted vs. manual data extraction

• In the controlled study, 13 CRCs extracted four randomly selected categories per protocol using AI-prefilled and blank spreadsheets; 97 tasks were completed.Source: §4.2 · p. 19

4.2.1 Efficiency of data extraction

• 40% faster: AI assistance reduced completion time by an average of 47 minutes per protocol and a median of 60 minutes (p = 0.008).Source: §4.2.1 · p. 20, Figure 2 · p. 20

4.2.2 Quality of data extraction

• AI-assisted tasks scored significantly higher in item-weighted accuracy and completeness than manual tasks (Hedges’ g = 0.53), using RAG 2 rather than the best RAG configurations.Source: §4.2.2 · pp. 20–21, Table 6 · p. 21

4.2.3 User satisfaction and preferences

• Of 12 surveyed CRCs, 11 found AI-assisted tasks easier and reported reduced thinking or decision-making; nearly all indicated likely future use.Source: §4.2.3 · p. 21

4.2.4 Other observations and examples

• Unassisted extraction showed three recurring error patterns: blank complex fields, generic non-informative responses, and direct transcription errors.Source: §4.2.4 · pp. 22–24

5 Discussion

• Across diverse therapeutic areas and document styles, specialized RAG matched or exceeded standalone LLM prompting, with its largest gains for scattered information and tabular Schedule of Events content.The discussion highlights adverse events, interventions, and site requirements among the categories with scattered information.Source: §5 · p. 25
• The pipeline uses the protocol itself as the source of truth for eligibility and a vision-enabled Schedule of Events workflow; its broader extraction scope is distinct from prior work, while finer eligibility structuring remains future work.The Claude Opus 4.5 configuration is described as comparable in performance while operationally simpler.Source: §5 · p. 25
• This work complements prior RAG applications by addressing structured extraction as a prerequisite to protocol generation and by covering a broader operational spectrum than narrower compliance-checking work.The system also provides element-level chunk references for auditability.Source: §5 · p. 25
• In the controlled CRC experiment, AI assistance reduced completion time, increased accuracy, was largely perceived as helpful, and lowered cognitive load.Source: §5 · pp. 25–26
• Standardized, traceable protocol representations may support downstream automation and decision support; the discussion frames efficiency and quality gains as potentially reducing burden and delays.Proposed uses include feasibility, start-up, site activation, and ongoing monitoring.Source: §5 · p. 26
• A small expert group cannot guarantee exhaustive ground truth because dense, dispersed information and subjective completeness make annotation vulnerable to oversight and fatigue, while reference verbosity may exceed operational needs.Source: §5 · pp. 26–27
• The scalable hybrid human–AI evaluation may inherit model biases and requires real-world validation; results may not generalize to some trials or specialized domains, and the modest dataset cannot capture full real-world variability.Sustained performance on out-of-distribution protocols will require ongoing validation and adaptation.Source: §5 · p. 27

6 Conclusion

• Quantified improvements in accuracy, completion time, and user preference support integrating clinical-trial-specific RAG extraction into research workflows, with expert review and continued validation.The authors describe the approach as promising for addressing trial complexity and supporting efficiencies across the trial lifecycle.Source: §6 · p. 28

Funding

• The authors report financial support from Banting Health AI Inc.Source: Funding · p. 29

Declaration of competing interest

• Francois Charest, Madison Wright, and Ramtin Babaeipour report Banting Health AI Inc. financial support and employment and equity or stock relationships.Source: Declaration of competing interest · p. 29
• The other authors declare no known competing financial interests or personal relationships that could have influenced the reported work.Source: Declaration of competing interest · p. 29

• The study-filtering logic starts from identifiable protocol documents listed in the Study Documents field and applies additional filters.Source: §A.1 · p. 35
• Drug interventions and locations in Canada or the United States are among the stated study filters.Source: §A.1 · p. 35
• Therapeutic-area classification uses disease or condition terms, including cancer-related terms and heart, cardiac, stroke, artery, coronary, cardio, and hypertension terms.Source: §A.1 · p. 35
• Scanned PDFs were filtered out so the final dataset contained text-based protocols processable by the RAG system.Source: §A.1 · p. 36
• Filtering intentionally excludes timestamps and other formatting or document-quality conditions; uniform random selection in each category supports diversity despite the modest dataset size.Source: §A.1 · p. 36
• Selected protocols vary in sponsors, phases, publication eras, structures, schedule-of-events formats, eligibility and endpoint formats, rendered HTML tables, diagrams, and amendment histories.Source: §A.1 · p. 36

A.2 Extracted semi-structured data models

• Intervention outputs capture arm and treatment details, including product, dose, schedule, administration, restrictions, and dose modifications.The example includes active PMA and saline placebo arms, with a dose reduction after local swelling greater than 50 mm.Source: §A.2 · pp. 36–37
• General-information and endpoint outputs combine protocol identifiers and study characteristics with objectives, endpoints, measurement methods, and endpoint timepoints.The example gives an 8-week endpoint timepoint and includes symptom and medication score measures.Source: §A.2 · pp. 37–38
• Schedule-of-events outputs represent visits through procedure lists, including recruitment, informed consent, immunotherapy administration, adverse-event assessment, and blood draws.One example visit is specified as Week 3 to 11.Source: §A.2 · pp. 38–39
• Eligibility outputs encode threshold-based inclusion criteria and exclusions, including symptom and skin-test thresholds and recent investigational-compound use.A separate comparison example highlights subtle data variation in an exclusion criterion.Source: §A.2 · pp. 39, 40, 44
• Adverse-event outputs structure definitions and severity, reporting timeframes, monitoring plans, and potential risks.The example specifies that an SAE form is completed immediately or within 24 hours of first onset or notification.Source: §A.2 · pp. 40–41
• Site-requirement outputs cover equipment and certifications, sample handling and access control, and investigational-product storage and accountability.The example includes refrigerated storage at 4 degrees Celsius and a daily working-day temperature log.Source: §A.2 · pp. 41–42
• A separate condition specifies receipt of any investigational compound within 30 days prior to screening.Source: §A.2 · p. 44
• Two entries specify allergen immunotherapy or SLIT within the past two years, with one specifying multi-antigen immunotherapy and prior-study or therapeutic-agent context.One entry also specifies receipt in a previous clinical study or as a therapeutic agent.Source: §A.2 · p. 44

A.3 Evaluation examples

• A submitted adverse-event definition that preserves the ground-truth wording and adds consistent, relevant detail retains the meaning without discrepancies.Source: §A.3 · pp. 44–45
• Omitting “perennial” from “perennial allergic rhinitis” leaves the core condition but loses specificity, earning a score of 4 for recovering at least 80% of ground-truth information.Source: §A.3 · p. 45
• Correctly reporting one site but omitting the ground-truth USA country information recovers less than 40% of the information and scores 2.Source: §A.3 · pp. 45–46

A.3.1 User satisfaction and preferences

• The user survey targets perceived usefulness, cognitive load, and adoption intent because extraction’s mental burden and time intensity matter for CRC retention and data quality.Source: §A.3.1 · p. 46
• Mental Demand and Effort items draw on NASA-TLX, while satisfaction and future-use items draw on TAM’s Perceived Usefulness and Perceived Ease of Use to compare workflows.Source: §A.3.1 · p. 46