PaperlayerView exact revision on arXiv

Source-linked AI summary

The Last Mile of Coding Agents for Long-Document Tasks: An Empirical Study on Clinical Trial Protocol Digitization

Jiacheng Lin, Geoffrey Low, Nishit Agarwal, Rahul Jain, Afrah Shafquat, Mandis Beigi, Jimeng Sun

arXiv:2610.18507v1cs.CL

TL;DR

It remains unclear whether coding agents can reliably transform complex long documents into accurate structured outputs, a consequential question for clinical trial protocols. The paper introduces TrialPD-Bench and evaluates coding agents, then develops TrialScribe; agents still fail in transcription and structuring, while TrialScribe improves a local model enough to match or surpass proprietary models.

  • Problem

    Whether coding agents can reliably transform complex documents into accurate, complete, structured outputs remains unclear, and protocol digitization matters because errors can affect downstream trial operations.

    Source: §1 · p. 1

  • Method

    The authors build TrialPD-Bench from 150 protocol PDFs, evaluate nine models in five coding-agent harnesses, and develop TrialScribe with task skills and a verification loop.

    Source: §1 · p. 2

  • Results

    Agents usually locate required evidence but fail in transcription and structuring; TrialScribe improves a local model by 6.2-59.4 points across eight metrics, enabling it to match or surpass proprietary models.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    Structured protocol representations are essential for making study requirements computable, interoperable, and reusable across the clinical trial life cycle.

    Source: §1 · p. 1

    Structured timeline extraction is evaluated on only three protocols, and downstream main- and sub-timeline extraction is outside the paper's scope.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Clinical trial protocols serve as the source of truth for study design, but are mostly written and used as unstructured PDF documents. Digitizing protocols turns static content into structured, machine-readable data that clinical software can analyze, validate, and execute. Coding agents, which select tools, execute code, and revise their approach as they work, are increasingly applied to such long-document tasks, yet it remains unclear how reliably they turn complete documents into accurate structured outputs. We study this question using protocol digitization as a testbed. We introduce TrialPD-Bench, an expert-annotated benchmark of 150 publicly available protocol PDFs spanning four task categories. Beyond final-answer performance, our evaluation measures evidence recall from agent traces, agent behavior, and cost. Across nine proprietary and open-weight models in five coding-agent harnesses, even the strongest agents do not fully solve protocol digitization. Our analysis further shows that agents usually locate the required evidence but fail in the last mile of transcribing and structuring it. We further develop TrialScribe, a domain-specific harness that equips coding agents with task skills and a verification loop. TrialScribe improves a locally served open-weight model by 6.2-59.4 points across eight metrics, enabling it to match or surpass proprietary models such as GPT-5.6-Sol.

1 Introduction

The paper studies whether coding agents can reliably turn long, complex documents into accurate structured outputs, using clinical trial protocol digitization as a high-stakes testbed. It introduces a benchmark and evaluation framework, then reports that agents still make substantial errors while TrialScribe improves a locally served model.

1 Introduction

  • Protocol digitization is a high-stakes long-document task because protocols combine heterogeneous layouts, complex tables, and information spread across sections, while errors can affect downstream operations.Structured representations support computable, interoperable, and reusable study requirements across the clinical trial life cycle.Source: §1 · p. 1
  • The evaluation measures evidence recall from agent traces, agent behavior, and cost in addition to final-answer performance.It evaluates nine proprietary and open-weight models in five coding-agent harnesses.Source: §1 · p. 2
  • TrialPD-Bench contains 150 complete protocol PDFs across four task categories, with structured outputs following a USDM-derived schema.The tasks cover disease indication, eligibility criteria, objectives and endpoints, and schedule of activities.Source: §1 · p. 2
  • TrialScribe improves Qwen3.8-27B by 6.2-59.4 points across eight metrics, enabling it to match or surpass proprietary models such as GPT-5.6-Sol.Even the strongest agents leave substantial errors, usually locating required evidence but failing in transcription and structuring.Source: §1 · p. 2

2 TRIALPD-BENCH: Evaluating Coding Agents on Protocol Digitization

TRIALPD-BENCH evaluates coding agents on four protocol-digitization tasks using full clinical-trial PDFs and expert annotations. Results show that evidence localization is often strong, but accurate relational extraction remains difficult and agents vary in behavior and cost.

2.1 Task Definition

  • The benchmark covers four structured extraction tasks from full protocol PDFs, with expert reference outputs and test results reported separately from TrialScribe development protocols.Tasks include disease indication, eligibility criteria, objectives and endpoints, and schedule of activities; five clinical-trial experts produced reference outputs.Source: §2.1 · p. 3, §2.2 · p. 3

2.3 Overview and Statistics

  • Protocols have a median length of 122 pages, vary in section and table structure, and can contain task-relevant content missing from the text layer.The corpus spans 18 therapeutic areas and 106 sponsors; 98 documents contain at least 100 pages.Source: §2.3 · pp. 3–4, Figure 2 · p. 3

2.4 Evaluation Metrics

  • Evaluation measures final-answer quality, evidence-page recall, agent behavior, and resource use, including vision use, tool errors, interaction turns, and cost.The reported tables cover final-answer metrics and evidence recall, behavior, vision use, and average run cost.Source: §2.4 · p. 4, Table 2 · p. 5, Table 3 · p. 5

2.6 Main Results

  • Best-agent scores range from 87.8% eligibility F1 and 86.8% SoA Activity-Visit F1 to 67.4% endpoint F1 and 23.2% encounter recall; relational tasks remain hardest.No final-answer metric exceeds 88% except timeline activity recall; the largest gaps are in relational components rather than surface content.Source: §2.6 · pp. 4–5, Table 2 · p. 5
  • Across tasks, agents generally visit required evidence pages, but evidence recall does not track answer quality; timeline encounter recall remains at or below 23.2% despite at least 98.3% page recall.Opus 4.6 has the highest SoA evidence recall at 96.0% but only 54.7% Activity-Visit F1; GPT-6-Astra has 84.8% recall and the best score, 86.8%.Source: §2.6 · pp. 5–6
  • SoA table extraction averages 39.8 turns and 2.18 tool errors per run, versus 7.1 turns and 0.55 errors for indication extraction; longer trajectories do not mean better answers.Sonnet 5 uses 69.4 turns for 74.7% SoA Activity-Visit F1, while GPT-6-Astra reaches 86.8% in 16.0 turns.Source: Table 3 · p. 5, §2.6 · p. 6
  • Comparable accuracy can come at markedly different costs: eligibility runs cost $0.40–$1.35 across frontier agents, while GPT-5.6-Sol is within 3.1 SoA F1 points of GPT-6-Astra at 40% of its cost.The SoA comparison is GPT-5.6-Sol at 83.7% and $0.85 per run versus GPT-6-Astra at $2.13.Source: Table 5 · p. 5, §2.6 · p. 6
  • Figure 3 plots SoA Activity-Visit F1 on complete protocols (Table 2) against model release date; lines connect models of the same family, and hollow markers denote open-weight models.Source: Figure 3 · p. 6

3 Further Analysis

Performance improved rapidly within model families but frontier agents remained below full accuracy, while failure modes varied by task and primarily reflected content, coverage, or structure errors rather than execution reliability.

3.1 Performance across Model Releases

  • Opus Activity-Visit F1 rose from 54.7% for Opus 4.6 to 86.0% for Opus 5, while frontier agents clustered at 83.7%–86.8% after July and open-weight models diverged.Kimi-K3 reached 83.9%, compared with 47.5% for DeepSeek-V4-Flash and 32.5% for Qwen3.8-27B.Source: §3.1 · p. 7

3.2 Failure Analysis

  • Eligibility failures were chiefly transcription errors (88.0%), indication failures domain grounding (51.7%), endpoint failures localization and coverage (26.7%), and SoA failures relational reconstruction (58.9%).For SoA, another 37.5% of failing runs missed annotated pages; one misresolved merged cell could shift later marks.Source: §3.2 · p. 7
  • Structured timeline failures were dominated by domain grounding (65.2%): agents often found visits but missed expected encounter categories and USDM v4.0 terminology.The analysis suggests agents recognize the standard but lack knowledge of its detailed specifications.Source: §3.2 · p. 7

3 Further Analysis

  • Execution reliability accounted for at most 3.6% of failing runs in any task, indicating that errors mainly concerned output content rather than producing usable output.The section examines model-release trends and failure types; further comparisons are reported in Appendix F.Source: §3 · p. 6, §3.2 · p. 7

4 TRIALSCRIBE: A Domain-Specific Harness for Protocol Digitization

TRIALSCRIBE addresses post-evidence errors with task-specific skills and verification; it substantially improves a locally served model, although complex table structure remains a gap.

4 TRIALSCRIBE: A Domain-Specific Harness for Protocol Digitization

  • The harness targets two obstacles: confidential protocols favor local open-weight models, while errors often persist after evidence is found.Its task skills supply domain knowledge and output conventions, and its verification loop audits outputs against the source.Source: §4 · p. 7

4.1 Design

  • Task skills encode protocol organization, domain knowledge, output conventions, and checks for completeness and source fidelity.Each skill is a SKILL.md file the agent reads at task start and applies as it works.Source: §4.1 · p. 7
  • A fresh read-only agent audits drafts against the protocol, reports source pages and corrections, and the agent applies minimal fixes for up to two rounds.A gold-free gate checks report schema validity and internal consistency; SoA extraction instead verifies within the same session.Source: §4.1 · p. 7

4.2 Results

  • TRIALSCRIBE improved Qwen3.8-27B across tasks, surpassing GPT-5.6-Sol on endpoint extraction (65.5% vs. 63.6%) and both proprietary agents on timeline activity and encounter recall.On timeline extraction, activity recall was 88.6% versus 88.1% and 87.2%, while encounter recall was 65.2% versus 23.2% and 0.0%; epoch accuracy remained below GPT-5.6-Sol, 60.0% versus 61.1%. The model was served locally using Pi on two NVIDIA A100 GPUs.Source: §4.2 · p. 8
  • Verification raised eligibility criteria F1 from 72.2% to 82.3%, objective F1 from 73.2% to 78.2%, and endpoint F1 from 62.0% to 65.5%, but left indication F1 at 77.5%.The pattern mirrors eligibility’s transcription-dominated failure profile, which source-grounded auditing can detect.Source: §4.2 · p. 8
  • On SoA tables, TRIALSCRIBE reached 75.1% Activity-Visit F1, below GPT-5.6-Sol at 83.7% and Opus 5 at 86.0%, leaving complex table reconstruction limited by the underlying model.It also reached 78.6% Sub/- sup F1.Source: §4.2 · p. 8

5 Related Work

Prior work spans protocol-specific extraction, document parsing, and coding agents, but most protocol digitization resources focus on one information type or use non-protocol text.

5 Related Work

  • Protocol digitization studies often target eligibility criteria or schedule-of-activities tables using registry text; Babaeipour et al. (2026) cover four categories on 23 unreleased protocols.Eligibility work includes Kang et al. (2017), Yuan et al. (2019), and Park et al. (2024); schedule-table work includes Dhuliawala et al. (2018) and Kumar et al. (2024).Source: §5 · p. 8
  • Coding agents and agent skills are increasingly applied beyond software engineering.The cited examples include Wang et al. (2024), Yang et al. (2024), Soni et al. (2026), and Li et al. (2026).Source: §5 · p. 8

6 Conclusion

The paper introduces TRIALPD-BENCH and finds that leading coding agents often fail after locating evidence; TRIALSCRIBE enables a locally served open-weight model to match or surpass proprietary agents.

6 Conclusion

  • The strongest coding agents usually locate required evidence but fail in the last mile of transcribing and structuring it.The paper introduces TRIALPD-BENCH as its benchmark for studying this gap.Source: §6 · p. 8
  • TRIALSCRIBE enables a locally served open-weight model to match or surpass proprietary agents such as GPT-5.6-Sol.Source: §6 · p. 8

Limitations

The findings are bounded by the protocol-digitization setting, although some may inform other long-document tasks; structured timeline extraction remains challenging and is evaluated on only three protocols.

Limitations

  • Findings are bounded by the use of clinical trial protocol digitization as the testbed for long-document tasks.The authors expect some findings to inform other settings, but do not claim broad generalizability.Source: Limitations · p. 9
  • Length, heterogeneous layouts, complex tables, and information distributed across sections are shared by other long documents, so some findings may inform those settings.The authors specifically expect the evidence-location versus transcription-and-structuring pattern to be informative beyond protocols.Source: Limitations · p. 9
  • Structured timeline extraction remains challenging, and its evaluation includes only three protocols.Extracting main and sub timelines is beyond the paper’s scope.Source: Limitations · p. 9

Ethics Statement

The paper reports no significant foreseeable ethical risks and describes coding agents as iterative systems combining a language model with a harness. Its related work spans protocol digitization, document understanding, and domain-supported coding agents.

Ethics Statement

  • The authors report no significant foreseeable ethical risks; the benchmark uses public ClinicalTrials.gov documents and author-produced annotations without external annotators or human-subject research.The documents were used as released, without collecting additional data.Source: Ethics Statement · p. 9

A.1 Coding Agents

  • A coding agent combines a language model with a harness that supplies tools and coordinates an iterative action–execution–feedback loop.The model’s behavior depends on both the model and the harness.Source: §A.1 · p. 12

B Additional Related Work

  • Related work covers protocol extraction, document parsing and structured extraction, and domain-supported coding agents; this work evaluates general-purpose agents and introduces TRIALSCRIBE’s domain skills and verification loop.The benchmark focuses on clinical protocol requirements, including activity–visit associations and objective–endpoint links.Source: B Additional Related Work · pp. 12–13

C Benchmark Details

TRIALPD-BENCH covers four task categories on 150 complete protocol PDFs, while prior resources generally focus on one information type or evaluate fewer protocols without releasing data. The benchmark uses expert-reviewed annotations and task-specific guidelines.

C.1 Comparison with Prior Work

  • TRIALPD-BENCH covers all four task categories on 150 complete protocol PDFs, whereas prior resources mostly target one information type and the comparable complete-PDF system evaluates 23 protocols without releasing its data.The prior system described by Babaeipour et al. covers all four categories.Source: §C.1 · p. 13

C.2 Corpus Construction

  • The corpus comprises protocols associated with 150 unique NCT identifiers, with each task using the subset that has reference annotations.A study can contribute to several tasks.Source: §C.2 · p. 13
  • The 150 documents total 17,471 pages, with a median of 122 pages and a range of 2 to 368; 16 documents have no extractable text layer.The corpus spans 18 FDA therapeutic-area categories, with two healthy-volunteer studies reported separately, and 106 sponsor labels.Source: §C.2 · p. 13

C.3 Annotation Process

  • Five clinical-domain experts produced and reviewed annotations, checking every AI-drafted field against the PDF; structured-timeline epochs, encounters, and activities instead use official CDISC example studies.The lead annotator made final decisions when annotations needed changes.Source: §C.3 · pp. 13–14
  • Task-specific guidelines define annotation rules for indication, eligibility, objectives and endpoints, and schedule-of-activities cells.Rules include verbatim transcription of eligibility criteria and resolving merged cells and multi-level visit headers.Source: §C.3 · p. 14

D Tasks and Evaluation Metrics

The appendix defines task outputs and evaluation metrics for protocol digitization, then compares coding agents with OCR and RAG baselines. Agents outperform both baselines on SoA extraction, while eligibility-category errors are rare.

D.1 Indication Extraction; D.2 Eligibility Criteria Extraction

  • Indication extraction returns protocol-worded diseases as separate entries, while eligibility extraction returns categorized criteria; both use F1, with eligibility matching normalized category-text pairs.Eligibility matching preserves repeated criteria, and a nonempty prediction against an empty reference scores zero.Source: §D.1 · p. 15, §D.2 · p. 15

D.3 Objectives and Endpoints Extraction

  • Objective and endpoint extraction preserves endpoint-to-objective links and levels; evaluation reports objective, endpoint, and objective-linked endpoint F1 separately.An endpoint assigned to the wrong objective fails the association metric even if its text is correct.Source: §D.3 · p. 16

D.4 Schedule of Activities Table Extraction

  • SoA extraction has a single-page transcription track and a complete-protocol track that adds page localization; both evaluate activity-visit relations and footnote-marker attachment.The document-level track includes 37 protocols and 70 reference tables, and scores every annotated SoA page, assigning zero to unrecovered pages.Source: §D.4 · pp. 16–17

D.5 Structured Timeline Extraction

  • Timeline evaluation covers epoch positional accuracy, semantic activity recall, and encounter recall requiring correct encounter type and contact modes.Epoch scoring does not realign sequences, so an inserted or omitted epoch shifts later comparisons.Source: §D.5 · p. 17

E.1 Coding Agents; E.2 RAG Baseline; E.3 OCR and Document-Understanding Baselines; E.4 Task Instructions

  • The evaluation uses expert annotations and isolated task runs, comparing nine models across coding-agent harnesses with RAG and OCR/document-understanding baselines under task-specific output instructions.Metrics are generally averaged over task instances; failed or missing outputs remain in prescribed denominators.Source: §E.1 · p. 17, §E.2 · p. 17, §E.3 · p. 18

F.1 Comparison with OCR and Document-Understanding Models

  • 84.3–90.0% Activity-Visit F1 and 85.3% Sub/sup F1 show coding agents exceeding the strongest specialist baselines on page-level SoA extraction.For document-level extraction, five of six strongest agents remain within 5 points of their page-level Activity-Visit F1.Source: §F.1 · p. 18, Figure 5 · p. 18

F.2 Comparison with a RAG Pipeline

  • 74.7–86.8% Activity-Visit F1 for the six strongest agents compares with 19.8–51.5% for RAG using the same models; agents also outperform RAG on other tasks.For nine of ten models, eligibility-criteria F1 improves by 14.3–55.9% and endpoint F1 by 4.7–28.9%.Source: §F.2 · pp. 18–19, Figure 6 · p. 19

F.3 Eligibility Criteria by Category

  • At least 99.5% category accuracy across every agent and RAG model indicates that eligibility-criterion F1 losses arise mainly from extraction, not classification.The appendix reports inclusion and exclusion F1 separately.Source: §F.3 · p. 19

F.4 Task-Level Results by PDF Text Accessibility

This section covers task-level results by PDF text accessibility, coding-agent failure patterns, and TRIALSCRIBE’s task skills and verification procedures. The supplied passages also describe a failure-profile figure but provide no task-level accessibility results.

F.4 Task-Level Results by PDF Text Accessibility

  • Tables 10–15 report agent metrics for all PDFs, text-sufficient PDFs, and vision-required PDFs, with accessibility defined separately for each task.A PDF is vision-required when task-relevant content is missing or unusable in its native text layer; the same protocol can fall into different groups for different tasks.Source: §F.4 · p. 19, Table 10 · p. 21

F.5 Full Numerical Results

  • Tables 16–19 provide exact scores for the results plotted in Figures 6 and 5 and for document-level SoA and structured timeline extraction.Table 16 includes linked endpoint F1, which is reported but not plotted; Table 17 covers 86 single-page inputs, while Tables 18 and 19 cover document-level SoA and three timeline example protocols, respectively.Source: §F.5 · p. 20, Table 16 · p. 24, Table 17 · p. 24

G.1 Failure Taxonomy

  • The failure taxonomy distinguishes transcription fidelity, localization and coverage, relational structure, domain grounding, and execution reliability.A run is classified as failing if it has at least one detected defect, regardless of score.Source: §G.1 · p. 20

G.2 Per-Model Failure Profiles

  • GPT-4o’s largest failure category is execution reliability in five of six tasks, while other agents have broadly similar within-task profiles; Table 20 provides additional examples.Source: §G.2 · p. 20, §G.3 · p. 20, Table 20 · p. 25

H.1 Development Protocols

  • TRIALSCRIBE’s skills and audit checklists were developed on small task-specific protocol sets that were excluded from the test split used for reported results.The development sets included eight protocols each for indication, eligibility criteria, and objectives and endpoints; one for SoA tables; and two for structured timelines.Source: §H.1 · p. 20

H.2.1 Indication Extraction; H.2.2 Eligibility Criteria Extraction; H.2.3 Objectives and Endpoints Extraction

  • TRIALSCRIBE’s text-extraction skills emphasize verbatim, complete source text and correct structure: disease indications, eligibility criteria, and endpoint-to-objective links.Indication extraction selects and normalizes the disease phrase; criteria extraction tracks list boundaries and preserves criterion granularity and encoding; objectives-and-endpoints extraction inventories both and links endpoints using source evidence rather than list position.Source: §H.2.1 · p. 21, §H.2.2 · pp. 21–22, §H.2.3 · p. 22

H.2.4 SoA Table Extraction; H.2.5 Structured Timeline Extraction

  • SoA extraction routes between text and rendered-page reading based on text-layer usability, then checks HTML structure and visit-column headers; timeline extraction inventories source evidence and preserves stages, encounters, activities, and timing.For timelines, missing evidence is null rather than a guessed default, and daily inpatient contacts and multiple contact modes are represented explicitly when supported by the protocol.Source: §H.2.4 · pp. 23–24, §H.2.5 · p. 24

H.3 Verification Loop

  • The verification loop uses an independent audit, a gold-free consistency gate, and source-supported fixes for up to two rounds; SoA tables instead use an in-session rendered-page check-and-revise loop.Audits identify affected fields, issues, corrections, source pages, and evidence; the gate checks report validity and candidate schema compliance.Source: §H.3 · pp. 24–25

F.4 Task-Level Results by PDF Text Accessibility

  • Figure 7 profiles dominant failure-category composition for ten coding agents across six extraction tasks, normalizing failing-run bars to 100% rather than showing failure rates.The ten agents comprise the nine main-text agents and GPT-4o.Source: Figure 7 · p. 32

H.4 Ablation of Skills and Verification

The ablation finds that both task skills and verification improve eligibility-criteria F1, with their combination producing the largest gain. A case study compares SoA extraction outputs, and future work identifies other protocol table types as a potential extension.

H.4 Ablation of Skills and Verification

  • On 40 eligibility-criteria protocols, the full TRIALSCRIBE gains 14.5 criterion-F1 points over the base agent; skills add 2.6 and verification 7.7 points alone.Verification adds 11.9 points with skills, versus 7.7 without them.Source: Table 27 · p. 33, §H.4 · p. 33

H.5 SoA Table Extraction Case Study

  • The SoA case study contrasts TRIALSCRIBE and six OCR or document-understanding baselines, with green, red, and pink cells marking correct, extra, and missing scheduling marks.Figure 10 tracks changes over successive verification passes.Source: §H.5 · p. 33

I Future Work

  • Other protocol table types used by downstream clinical systems are outside the current coverage, and extending the benchmark and harness to them is identified as future work.Table 28 lists common examples.Source: I Future Work · p. 33

K Data Sources and Licenses

The benchmark’s protocol PDFs are publicly posted on ClinicalTrials.gov, but are not redistributed. Instead, users receive identifiers and URLs to obtain the documents from the registry.

K Data Sources and Licenses

  • TRIALPD-BENCH does not redistribute protocol PDFs; it provides each document’s NCT identifier and ClinicalTrials.gov URL so users can obtain it from the registry.Sponsors upload the protocols as part of results reporting.Source: K Data Sources and Licenses · p. 33
  • The structured timeline references come from CDISC’s published USDM example studies, while the reference annotations and code are released under CC BY 4.0.The CDISC examples are used as released and credited to CDISC.Source: K Data Sources and Licenses · p. 33

Related papers

Loading 2610.18507v1…