PaperlayerView exact revision on arXiv

Source-linked AI summary

GxP-Agent: Process-DAG Topology for Reliable Clinical Trial Programming with LLM Agents

Jaime Yan

arXiv:2608.16890v1cs.AI

TL;DR

Clinical trial programming remains difficult for LLMs, with single-shot generation failing on ADSL. GxP-Agent encodes regulatory process ordering as a DAG of specialized agent tasks, achieving 100% structural match with Claude versus 59.2% for the best retrieval baseline and 0% for single-shot approaches.

  • Problem

    Single-shot LLM generation achieves a 0% pass rate for ADSL, motivating whether predefined dependency structure should guide agent decomposition.

  • Method

    GxP-Agent encodes regulatory process ordering as a directed acyclic graph that assigns derivation steps to specialized worker agents.

  • Results

    100% structural match was achieved with Claude on ADSL, compared with 59.2% for the best retrieval baseline and 0% for single-shot approaches.

  • Takeaways & Limitations

    Encoding known derivation decomposition and ordering as graph topology can support reliable clinical trial programming with LLM agents.

  • Takeaways & Limitations

    Generalization beyond CDISCPilot01 and its ADSL/ADAE tasks remains untested, and stronger retrieval baselines might narrow the reported gap.

Abstract

from arXiv · show

Clinical trial programming -- transforming study protocols into analysis-ready datasets under CDISC standards -- is a bottleneck in regulatory submissions, yet LLM-based code generation fails catastrophically on this task: across 11 single-shot attempts with five frontier models, none produces a valid subject-level analysis dataset. We introduce GxP-Agent, a multi-agent system that encodes regulatory process ordering as a directed acyclic graph (DAG), decomposing monolithic dataset generation into 15 domain-specific nodes executed by worker agents with pharmaverse skill context, validation gates, and conditional retry. On CDISC-Bench, a new execution-based benchmark built from the FDA pilot submission CDISCPilot01 (254 subjects, 49 ground-truth ADSL variables), GxP-Agent with Claude Sonnet 4.6 achieves 100% structural match (49/49 variables, 254 correct records) across three independent runs, compared to 59.2% for the best retrieval-augmented baseline and 0% for all single-agent and flat multi-agent approaches. The DAG topology also enables weaker models: GPT-4.1 achieves 59.2% mean structural match under the same DAG, where it scores 0% under every other architecture. The approach generalizes to ADAE (adverse events; 9-node branching DAG, 55 variables, 1,191 records), achieving 100% structural match on the first attempt. These results demonstrate that encoding domain process knowledge as graph topology -- rather than relying on LLM reasoning alone -- is a key enabler for reliable, GxP-compliant clinical trial programming.

1 Introduction

Clinical trial programming is a labor-intensive, GxP-constrained process that frontier LLMs fail to automate reliably in single-shot generation. GxP-Agent addresses this by encoding regulatory process ordering as a DAG, supported by CDISC-Bench and evidence that topology improves performance, especially for weaker models.

  • Motivation: Clinical trials support approximately 80% of FDA drug approvals, while producing CDISC ADaM datasets and TLFs requires weeks of specialized programming under GxP compliance.The workflow transforms raw clinical data into analysis-ready datasets and produces Tables, Listings, and Figures according to the Statistical Analysis Plan.
  • Motivation: 0% pass rate was achieved across 11 single-shot ADSL attempts spanning five frontier models.Manual review identified recurring failures including variable-name hallucination, function-signature hallucination, incorrect file references, and logic errors.
  • Contribution: GxP-Agent encodes regulatory process ordering as a directed acyclic graph, assigning derivation steps to specialized worker agents.Its system includes node-type-specific prompts, validation gates, schema introspection, and conditional retry.
  • Benchmark: CDISC-Bench evaluates executable clinical-programming outputs using FDA CDISCPilot01 ground truth containing 254 ADSL records and 49 variables.The benchmark uses five evaluation levels.
  • Results: 100% structural match was achieved with Claude, 59.2% with GPT-4.1, and 0% for all non-DAG architectures except Keyword-RAG+Sonnet at 59.2%.The Claude result covered one Opus and three Sonnet runs with 0% variance; DAG decomposition also disproportionately benefits weaker models by isolating failures to individual nodes.

2 Background

CDISC separates standardized clinical data from analysis-ready datasets, with ADSL as the foundational subject-level dataset requiring coordinated derivations across SDTM domains. Pharmaverse tools support clinical-trial reporting, while graph-based agent workflows provide conditional execution mechanisms; the proposed approach fixes decomposition and ordering through domain knowledge rather than runtime LLM planning.

  • CDISC standards: ADSL is the foundational ADaM dataset, containing one record per subject and requiring derivations coordinated across multiple SDTM domains.The passage identifies treatment dates, disposition, vital signs, and questionnaire scores as examples of cross-domain derivations.
  • Pharmaverse: Pharmaverse provides open-source R packages for clinical-trial reporting, including admiral, metacore, metatools, and xportr.Their stated functions cover ADaM derivation, metadata specifications and utilities, and XPT export for regulatory submission.
  • LLM agents and graph orchestration: Graph frameworks such as LangGraph support agent workflows with state machines, conditional edges, and retry logic, but the proposed approach predetermines decomposition and ordering using domain knowledge.This contrasts with existing multi-agent systems that rely on LLM reasoning for task decomposition and can encounter planning failures.

3 System Design

GxP-Agent operationalizes clinical-trial programming as executable process DAGs: a manager selects a template, workers execute node-specific steps, and validation gates enforce domain constraints. The ADSL workflow uses incremental, topologically ordered derivations, while the topology guarantees completeness, dependency ordering, failure isolation, and auditability.

  • Architecture: The Project Manager Agent selects one of 11 predefined process-DAG templates and compiles the chosen task into an executable graph.The current registry contains 11 templates, with ADSL evaluated here.
  • Architecture: The DAG Compiler maps each process step to a LangGraph StateGraph agent invocation with conditional success routing, up to 2 retries, and skip-after-exhaustion behavior.Prompts incorporate node type, description, and pinned skills; workspace files and schemas are inspected before execution.
  • Execution and validation: Worker Agents execute nodes with R execution, schema inspection, and file-management tools, while post-execution checks verify expected output columns.Validation Gates additionally apply 12 ADSL-specific R assertions covering record-level, variable-level, and business-rule-level constraints.
  • ADSL process DAG: The ADSL process DAG contains 15 topologically ordered nodes, with each node loading the prior .rds file and adding only 2–5 derived variables.This incremental construction avoids requiring each worker to derive all 49 variables simultaneously.
  • Structural guarantees: The DAG topology guarantees completeness, dependency-respecting ordering, failure isolation, and traceable node artifacts for GxP compliance audit trails.The framework supports both linear workflows such as ADSL and branching workflows such as ADAE.

4 CDISC-Bench

CDISC-Bench is an execution-based benchmark built from the FDA’s CDISCPilot01 submission to evaluate clinical trial programming systems. Its evaluation emphasizes execution and structural correctness, while spot checks assess selected computed values separately.

  • Benchmark construction: CDISC-Bench uses CDISCPilot01, which includes 306 enrolled subjects, 254 intent-to-treat subjects, and 3 treatment arms.The benchmark is based on the FDA’s publicly available CDISC pilot submission.
  • Benchmark construction: 22 SDTM domain .xpt files provide the tabulated clinical source data for benchmark tasks.The source domains include DM, AE, EX, DS, LB, VS, SC, QS, SV, and MH, among others.
  • Benchmark construction: Ground-truth outputs include ADSL with 254 records and 49 variables, ADAE with 1,191 records, and ADLBC with 7,778 records.The pilot submission also provides an ADaM Specifications document defining variable derivation logic, source domains, business rules, and controlled terminology.
  • Evaluation framework: The five-level framework increases in stringency, with the paper focusing on L3 execution and L4 structural match.Together, L3 and L4 determine whether the generated dataset is structurally correct.
  • Evaluation framework: 100% structural match means all 49 expected ADSL variable names are present, but does not guarantee value correctness.L5 spot checks therefore verify placebo arm count 86, mean age in the placebo arm 75.21, and ITT flag count 254.

5 Experiments

Experiments show that DAG topology, combined with validation and retry, converts otherwise unsuccessful clinical-trial programming attempts into reliable structural dataset generation. Across ADSL and ADAE, the approach achieves strong structural matches, while remaining errors concentrate in specific derivations and numeric precision.

  • ADSL results: 0% of 16 single-shot attempts produced a valid ADSL across five models.The attempts included 11 base runs plus retry and metadata variants.
  • ADSL results: 100% structural match was achieved by Claude Sonnet under DAG, versus 0% under SingleAgent, FlatMulti, and Embedding-RAG.Embedding-RAG retrieved 15 relevant skill documents; Fisher’s exact test for DAG versus non-DAG gives p = 0.011.
  • ADSL results: 59.2% structural match was achieved by GPT-4.1 under DAG, versus 0% under every non-DAG architecture.The same DAG enabled a weaker model to succeed where alternative architectures failed; Cliff’s d = 0.267 for Claude versus GPT-4.1 node completion.
  • Reliability and validation: 100% was achieved in all four Claude DAG runs, with correct value spot-checks and 6/45 Sonnet node-executions requiring a second attempt.The DAG evaluation used identical topology, prompts, retry logic, and validation while invoking models directly per node to isolate topology effects.
  • Node-level analysis: 100% success was observed for ingest and merge_dm, while derive_disposition, derive_completion, and downstream validation nodes exposed model-specific or cascade-sensitive failures.Reported failures include GPT-4.1 at 0/3 on derive_disposition and GPT-4o at 0/3 on derive_completion.
  • ADAE generalization: 100% structural match was achieved on ADAE in both Claude Sonnet runs, covering 55/55 variables and 1,191 records.ADAE used a nine-node branching DAG and one-record-per-event BDS structure; systematic L5 evaluation reached 67.6% column accuracy (23/55 exact).
  • Node-level analysis: 74.5% of DAG v5 node executions completed successfully, including 100% for Claude, 73.3% for GPT-4.1, and 48.9% for GPT-4o.Claude spot-checks matched ground truth for placebo_n=86, mean_age=75.21, ITTFL_Y=254, and SAFFL_Y=254.
  • Value-level analysis: 73.7% mean column accuracy across Claude runs showed systematic residual errors concentrated in floating-point rounding and complex multi-step conditionals.Identifiers and demographics were universally correct, while BMIBL, HEIGHTBL, COMP8FL, and DCSREAS accounted for concentrated failures.

6 Ablation Studies

The ablations show that architecture and runtime context jointly determine reliable dataset generation, while model tier and retry logic further affect node completion. DAG topology alone is insufficient without sufficiently specific prompts and schema introspection.

  • A1: Architecture: 100% versus 0%: Claude Sonnet succeeds only under the DAG, while GPT-4.1 reaches 59.2% only under the DAG architecture.Claude Sonnet scores 0%/0%/59.2%/100% under SingleAgent/FlatMulti/Keyword-RAG/DAG; GPT-4.1 scores 0%/0%/0%/59.2%.
  • A2: Prompt specificity: 0% structural match: generic prompts achieve 73% mean node completion but fail because validation merges incorrect intermediate artifacts.These runs use the same 15-node ADSL DAG but omit node-type hints, skills, and specification text; the validate node yields 0 records.
  • A3: Schema introspection: 42.9% (21/49 variables): schema introspection raises structural match from 0% while holding topology and prompts constant.Without schema introspection, Sonnet completes 8/9 nodes but uses wrong column references; GPT-4.1 completes 6/9 nodes without it and 5/9 with it, both at 0%.
  • A4: Model tier: 100% node completion: Claude models form the top tier on the fixed DAG, while Gemini completes 53.3% individually and fails on validation.GPT-4.1/4o succeed on simple nodes but fail on derivations requiring precise dplyr idioms.
  • A5: Retry: 74.5% total: retry recovers 16 additional nodes beyond the 64.8% first-attempt rate.Retry lifts Claude Sonnet from 86.7% to 100% (+13.3 pp) and GPT-4.1 from 53.3% to 73.3% (+20.0 pp), but provides no lift for GPT-4o.

7 Discussion

The DAG succeeds by reducing prompt complexity, isolating node failures, and targeting prompts for a 49-variable, 8-domain ADSL task. It also outperforms retrieval baselines and preserves inspectable artifacts for GxP-aligned recovery when failures occur.

  • Why DAG topology works: ∼2K-token node prompts, versus ∼15K for monolithic specifications, reduce context while targeted prompts support anti-hallucination guidance.The DAG also preserves upstream artifacts when a later node fails, enabling retry without upstream loss.
  • Why DAG beats retrieval baselines: 59.2% (29/49 vars) is achieved by Keyword-RAG+Sonnet, while the DAG reaches 49/49 and enables GPT-4.1 to match 59.2%; Embedding-RAG+Sonnet scores 0%.The comparison indicates that contextual augmentation alone does not decompose the 49-variable, 8-domain task.
  • Failure modes: When the DAG fails, completed nodes preserve .rds artifacts and per-node logs, allowing users to fix the failing node and resume.This produces partial, inspectable results aligned with GxP audit trail requirements, unlike single-shot failure, which produces no usable output.

8 Related Work

Prior work spans general LLM code-generation benchmarks, multi-agent software systems, and clinical or scientific domain-specific code generation. GxP-Agent addresses a specialized compliance domain through predetermined, domain-knowledge-driven DAG decomposition, with the paper emphasizing structural decomposition over contextual augmentation.

  • LLM-based code generation: General programming benchmarks evaluate LLM code generation, while SWE-agent and self-debugging improve quality through agentic interfaces and iterative refinement.These benchmarks do not capture the unique challenges of specialized clinical-trial compliance.
  • Multi-agent architectures: Multi-agent systems support software development, scientific discovery, and data analysis, with systems such as MASAI and CodeR decomposing tasks through sub-agents or task graphs.Agentless further shows that structured pipelines can match agentic approaches.
  • Multi-agent architectures: GxP-Agent differs by using predetermined, domain-knowledge-driven decomposition in which each DAG node maps to a concrete derivation step.This topology is presented as distinct from general-purpose multi-agent decomposition.
  • Clinical NLP and domain-specific code generation: Clinical automation has addressed protocol understanding, note processing, outcome prediction, and TLF generation from ADaM data, while scientific code-generation studies show domain context can improve quality.The paper’s results suggest structural decomposition matters more than contextual augmentation through skills or retrieval.

9 Limitations

The study’s findings are limited by unresolved prompt effects, potentially stronger retrieval baselines, narrow dataset coverage, possible model-training exposure differences, and substantially higher execution cost. Limited run counts also constrain confidence in the reported results.

  • Methodological limitations: The full DAG’s 0%→100% topology-only ablation gradient does not disentangle its multiple prompt choices through factorial design.The ablation demonstrates a gradient, but cannot isolate the contribution of individual prompt choices.
  • Methodological limitations: Stronger retrieval baselines might narrow the reported performance gap.
  • External validity: Generalization remains untested beyond CDISCPilot01 and its ADSL/ADAE datasets.All experiments use CDISCPilot01.
  • External validity: Claude’s 100% versus GPT-4.1’s 59% may partly reflect differential training-data exposure rather than architecture alone.
  • Computational cost: 662s versus 65s makes the DAG substantially slower than Keyword-RAG because it requires 15 sequential LLM calls.
  • Statistical limitations: Only 1–3 runs per condition limit confidence in the stability of the reported results.

10 Conclusion … D Extended DAG Version Changelog

GxP-Agent uses regulatory process ordering encoded as DAG topology to improve clinical trial programming reliability. The supplied sections document its ADSL benchmark results, 15-node process structure, 49-variable ground truth, per-node evaluation, and early DAG-version changes.

  • 10 Conclusion: 100% structural match was achieved on ADSL with Claude across 1 Opus + 3 Sonnet runs, with 0% variance.All single-shot approaches scored 0%, while the best retrieval-augmented baseline achieved 59.2%.
  • A ADSL DAG Node Descriptions: The ADSL process DAG contains 15 nodes whose descriptions specify node types, input SDTM domains, and output variables.These descriptions are provided in Table 6.
  • B CDISC-Bench Ground-Truth Variables: 49 variables comprise the ADSL ground truth, organized in Table 7 by derivation category.The list contains all variable names used for the benchmark ground truth.
  • B CDISC-Bench Ground-Truth Variables: Exact name matches against the 49-variable ground-truth list define the structural match metric.The “Other” category includes variables appearing in the CDISCPilot01 ground-truth XPT file.
  • C Per-Node Success Rates: Per-node success rates were reported across 11 DAG v5 runs spanning Opus, Sonnet, GPT-4.1, GPT-4o, and Gemini.Table 8 summarizes these per-node rates across 1 Opus, 3 Sonnet, 3 GPT-4.1, 3 GPT-4o, and 1 Gemini runs.
  • D Extended DAG Version Changelog: v1 used 9 nodes without schema injection; Sonnet completed 8/9 nodes, but validation failed and Struct% = 0%.The failure involved missing variables.
  • D Extended DAG Version Changelog: v2 added upstream .rds column names to worker prompts; Sonnet completed 8/9 nodes and reached Struct% = 42.9% (21/49 vars).Schema context eliminated wrong-column-reference errors.
  • D Extended DAG Version Changelog: v3 tested GPT-4.1, which completed 6/9 nodes and reached Struct% = 0%.The version identified GPT-4.1-specific hallucination patterns involving admiral function signatures.

E Additional Limitations

The evaluation lacks a qualified statistical programmer baseline and relies on aligned row ordering for systematic value comparison. These limitations prevent assessment of the practical human–LLM performance gap and may undercount true matches when rows are misaligned.

  • Additional limitations: No human baseline prevents comparison with qualified statistical programmers on the same task.The study therefore cannot assess the practical gap between LLM-generated and human-written code.
  • Additional limitations: Aligned row ordering is assumed for systematic value comparison, so misaligned rows may undercount true matches.Cross-run L5 variance analysis is deferred.
Loading 2608.16890v1…