PaperlayerView exact revision on arXiv

Source-linked AI summary

ProCARE: Real-World Study Automation via Profile-Grounded Evidence Contracts

Zhaoyu Fan, Bowen Han, Jincheng Ou, Yanwei Ren, Zehua He, Kaiyu Huang, Pengcheng He, Shanshan Ni, Chengchen Gong, Shuai Xue, Qingjiang Shi

arXiv:2610.16175v1cs.AI

TL;DR

RWS workflows must preserve relational structure, clinical variable semantics, temporal anchors, analysis grain, and evidence scope, which standard LLM-agent interfaces do not fully preserve for schema-rich clinical data. ProCARE profiles heterogeneous data, compiles study goals into typed RWS-IR, validates generated artifacts with deterministic checks and localized repair, and admits claims through evidence certificates. ProCARE improves executable validity, code quality, and claim traceability relative to LLM-agent and table-analysis baselines.

  • Problem

    RWS workflows must preserve relational structure, clinical variable semantics, temporal anchors, analysis grain, and evidence scope, which standard LLM-agent interfaces do not fully preserve for schema-rich clinical data.

    Source: §1 · p. 1

  • Method

    ProCARE profiles heterogeneous data, compiles study goals into typed RWS-IR, validates generated artifacts with deterministic checks and localized repair, and admits claims through evidence certificates.

    Source: §1 · p. 2, §3 · p. 3, §3.1 · p. 3

  • Results

    ProCARE improves executable validity, code quality, and claim traceability relative to LLM-agent and table-analysis baselines.

    Source: §6 · p. 10

  • Takeaways & Limitations

    ProCARE is a contract layer for making RWS assumptions, artifacts, and claim support auditable under human oversight.

    Source: §5.2 · p. 9

    Reliability depends on profile correctness, validator coverage, and RWS-IR assumptions, while the primary benchmark is single-center and disease-specific.

    Source: §5.2 · p. 9

Abstract

from arXiv · show

Real-world studies (RWS) analyze routinely collected healthcare data, such as electronic health records and insurance claims, to estimate treatment effects, monitor safety, and identify risk factors outside randomized trials. Automating RWS with large language model agents is appealing but brittle: valid analyses depend on dataset-specific clinical semantics, including index dates, follow-up windows, patient/event-level units, and outcome boundaries that are often implicit in observational data. This leads to schema mismatches, temporal errors, inconsistent analytical granularity, and unsupported clinical claims. We present ProCARE, a profile-grounded framework that reformulates RWS automation as constrained compilation. ProCARE converts heterogeneous data profiles into Evidence Contracts specifying available variables, temporal attributes, distributions, and evidentiary limits, and compiles study objectives into RWS-IR, a strongly typed intermediate representation for variables, time relations, analysis units, and statistical tasks. Deterministic validation and localized repair then check and revise generated workflows across schema mapping, cohort construction, variable derivation, analysis, and interpretation. On 36 EHR-based benchmark tasks, ProCARE outperforms strong LLM-agent baselines in code executability, semantic consistency, and conclusion traceability, with automated violation diagnoses showing moderate-to-substantial agreement with blinded expert review.

1 Introduction

RWS automation must preserve clinical, relational, temporal, and evidentiary constraints that generic data-analysis workflows do not capture. ProCARE addresses this by compiling profile-grounded study specifications into validated workflows and evidence-linked reports.

1 Introduction

  • RWS workflows must preserve relational structure, clinical variable semantics, temporal anchors, analysis grain, and evidence scope across the analysis and report.Cohorts span heterogeneous tables, and reported findings must remain within the support of the executed analysis.Source: §1 · p. 1
  • Textualizing multi-table data can obscure entity keys, visit structure, table linkage, missingness, and time-dependent variables.RWS also requires reasoning about endpoints, censoring, confounding, proxy variables, and claim scope [8].Source: §1 · pp. 1–2
  • Three recurring failures occur at query-to-specification, specification-to-code, and artifact-to-claim transitions.They include incompatible field bindings, violations of joins or temporal ordering, and unsupported clinical claims from descriptive or proxy artifacts.Source: §1 · p. 2
  • ProCARE uses a heterogeneous profile as an evidence contract, compiles goals into typed RWS-IR, and validates generated artifacts with deterministic gates and localized violations.Claims are admitted only through certificates linking them to validated artifacts, executed code, profile entries, or recorded assumptions [10].Source: §1 · p. 2
  • The paper evaluates ProCARE on 36 HCC tasks and 18 public-health transfer tasks across executability, code quality, artifact completeness, and claim traceability.Its stated contributions include a heterogeneous profile and typed RWS-IR, plus evidence certificates that constrain factual report claims.Source: §1 · p. 2

2 Related Work

Prior work advances LLM workflow automation and structured-data grounding, while RWS adds study-level constraints on design, execution, and claims. ProCARE positions its contract as spanning those requirements.

LLM agents for data analysis

  • Data-analysis agents translate natural-language goals into programs and outputs, but reported failures include state tracking, dependency management, numerical consistency, and data-dependent repair.In RWS, workflows must also preserve relational structure, analysis grain, temporal anchors, clinical roles, and claim scope [11] [12] [13] [14] [15] [16] [18].Source: §2 · p. 2

Structured and clinical-data grounding

  • Table-centric, text-to-SQL, structured-EHR, and FHIR approaches improve structured-data access but mainly target retrieval, query answering, or task-level execution.RWS requires variable bindings, joins, temporal anchors, artifacts, and claims to remain valid under one study-level contract [19] [21] [23] [24] [25] [26].Source: §2 · p. 2

RWS evidence constraints

  • RWS evidence constraints include transparent reporting, explicit protocols and assumptions, standardized representation, and reliable provenance.ProCARE incorporates these constraints through typed RWS-IR, localized execution violations, and evidence certificates for factual claims.Source: §2 · p. 2
  • Figure 1 depicts a heterogeneous profile constraining RWS-IR compilation, violation-guided execution, and evidence-certified DAG reporting.The architecture presents these stages as a profile-grounded evidence contract.Source: Figure 1 · p. 3

3 Method

ProCARE propagates a heterogeneous data profile through typed study specification, constrained execution, and evidence-certified reporting. Static and runtime checks localize unsupported choices and violations before results and claims are admitted.

3 Method

  • ProCARE treats RWS automation as three profile-constrained transitions from goal to specification, specification to validated code, and artifacts to admissible claims.Study choices, code operations, and claims must trace to profile entries, validated artifacts, or explicit assumptions.Source: §3 · p. 3

3.1 Heterogeneous Data Profiling

  • The heterogeneous profile separates dataset/task and field/instance levels from structural and knowledge-driven abstractions.Structural blocks specify joins, grain, temporal anchors, and artifact constraints; knowledge blocks record clinical roles, ontology mappings, proxy status, assumptions, and claim scope. Ambiguous mappings are marked semantically unknown, limiting them to structure-grounded operations unless a proxy or assumption is recorded.Source: §3.1 · p. 3, Table 1 · p. 4

3.2 RWS-IR Compilation and Violation-Guided Execution

  • Typed RWS-IR encodes study design, analysis grain, joins, keys, temporal anchors, variable roles, methods, artifacts, and claim policy.Static checks require profiled fields and keys, supported or explicitly assumed clinical roles, and compatible joins, index dates, follow-up windows, and grain; unsupported IRs are repaired, downgraded, or reported infeasible. Execution gates then check schema, methods, and artifacts, and localized violations guide repair; unsupported tasks may terminate with a failure trace or be downgraded to descriptive analysis.Source: §3.2 · p. 4

3.3 Certified DAG-Based Reporting

  • Evidence certificates admit claims only when they are traceable to validated evidence and permitted by the RWS-IR claim policy.Unsupported claims are removed or rewritten as limitations; proxy-based or descriptive analyses may support descriptive or association claims, but not causal, safety-critical, or high-stakes claims without required assumptions and validations. A dependency DAG orders report components by prerequisites, while certificates govern claim admissibility.Source: §3.3 · pp. 4–5

4 Evaluation and Benchmark

The evaluation tests complete RWS artifact bundles on a de-identified longitudinal HCC benchmark and a public-data transfer suite, using deterministic checks and blinded rubric-based judges. It also separates artifact-level comparisons from contract-native diagnostics and notes that profile-derived validity constraints leave some evaluator coupling.

Evaluation protocol

  • The benchmark scores complete artifact bundles, with deterministic validators checking executability, schema use, completeness, temporal/grain consistency, and claim support.Rubric-based judges assess residual semantic ambiguity, presentation quality, and stylistic criteria; they are blinded to method identity.Source: §4 · p. 5

Output contract

  • Each expected bundle includes a study specification, executable entry point and artifacts, and a structured report with assumptions, limitations, and evidence links.Hallucinated tables or columns are disallowed; prompt-specific instructions govern feasibility and handling of missing required inputs.Source: §4 · p. 5

Evaluation metrics

  • Execution-time violation resolution is measured separately from final-report claim certification, unsupported claims, and language violations.The supplied metric description identifies these measurement scopes but does not provide enough information here to report formula values.Source: §4 · p. 5

Scoring protocol

  • Evaluation mitigates but does not eliminate evaluator coupling because validity remains operationalized through profile-derived constraints.Native contract artifacts are treated as system outputs rather than ground truth, while non-ProCARE specifications and claim links are reconstructed post hoc where possible.Source: §4 · p. 6

Primary HCC benchmark

  • The primary benchmark comprises 1,295 subjects, 12,847 visits, and 54 linked electronic case-report-form sheets from a de-identified longitudinal HCC cohort.Its tables retain subject- and visit-level structure for repeated-measures and time-to-event analyses.Source: §4 · p. 6

Public-RWS external validation suite

  • Public-RWS-18 adds 18 tasks across three public datasets to test schema shift, multi-table grounding, proxy-outcome handling, and claim traceability.The datasets are UCI Diabetes 130-US Hospitals, a NHANES subset, and TCGA-LIHC.Source: §4 · p. 6

5 Experiments

Across 36 primary HCC tasks, ProCARE’s contract mechanisms support executable, clinically grounded workflows and more traceable claims. Robustness analyses examine prompt specificity, public-data schema transfer, backbone sensitivity, and profiling costs, while the authors bound the conclusions to auditable checks under human oversight.

5 Experiments

  • The evaluation spans 36 primary HCC tasks and 18 Public-RWS external-validation tasks, but the authors characterize the primary benchmark as single-center and disease-specific.Public-RWS-18 mainly tests schema transfer rather than multi-institutional longitudinal generalization; deployment on other schemas requires re-instantiating profiles and revalidating checks.Source: §5 · p. 6, §5.2 · p. 9

5.1 Main Results

  • ProCARE achieves the strongest violation resolution and post-hoc claim traceability among the scaffolded controls, which improve generic baselines in executability and artifact completeness.The comparison uses evaluator-facing information, output schemas, validator feedback, and repair budgets without transplanting ProCARE’s core modules.Source: §5.1 · p. 7, Table 4 · p. 7
  • Ablations attribute the largest contribution to the profile, with RWS-IR, violation-guided repair, evidence certificates, and certified DAG ordering serving complementary roles.Structural contracts support computation, repair improves execution robustness, and evidence and ordering contracts regulate reporting.Source: §5.1 · p. 7, Table 5 · p. 7
  • More specific prompts improve CTC and reduce UCR/CLV, mainly by constraining variable bindings, temporal anchors, and methods.Remaining claim downgrades mostly concern causal or high-stakes wording unsupported by proxy artifacts, sparse endpoints, or subgroup evidence.Source: §5.1 · pp. 7–8, Table 6 · p. 8
  • Automated diagnostics show moderate-to-strong Spearman agreement with blinded expert review, strongest for RWS-IR validity (ρ = 0.82).The review involved three clinical and two statistics/data experts; correlations used 36 primary HCC instances, with all p < 0.001.Source: §5.1 · p. 8, Table 7 · p. 8

5.2 Robustness and Deployment Analyses

  • Across four HCC RWS objectives, performance is generally better with more specific prompts and stable across task families; public-data transfer favors ProCARE over Claude Code and Scaffolded G3P on four diagnostics.NHANES remains hardest because of multi-table survey joins; claim policies limit outputs to dataset-specific descriptive, proxy, or association claims.Source: §5.2 · p. 8, Figure 2 · p. 8, Table 8 · p. 9
  • With the profile, RWS-IR, validators, and certificate gate fixed, ProCARE remains contract-consistent across tested backbones, while stronger planners mainly improve GA, PF, VRR, and CTC.Execution metrics are comparatively stable; full backbone results are in the appendix.Source: §5.2 · p. 8, Table 9 · p. 9
  • At 20 queries on one dataset, ProCARE’s amortized cost is about 41.3k tokens, versus about 103k tokens for one query.Profiling costs about 65k tokens once per dataset and inference about 38k tokens per query; baseline token budgets are not directly comparable because their pipelines differ.Source: §5.2 · p. 9

6 Conclusion

ProCARE improves executable validity, code quality, and claim traceability relative to LLM-agent and table-analysis baselines. The supplement documents benchmark scope, schema, release boundaries, and worksheet organization.

6 Conclusion

  • ProCARE improves executable validity, code quality, and claim traceability relative to LLM-agent and table-analysis baselines.The conclusion reports results on the 36-instance HCC benchmark and Public-RWS-18 external validation suite.Source: §6 · pp. 9–10

A Appendix Roadmap, Navigation, and Main-Paper Coverage

B Benchmark Datasets and Tasks

  • The benchmark-details section covers release contents, worksheet organization, task instantiation, public-data transfer tasks, and validator worksheet inventory.Source: B Benchmark Datasets and Tasks · p. 15

B.1 Supplemental schema scope

  • The workbook spans enrollment, baseline characterization, time-varying interventions, follow-up, and survival outcomes, shaping join, grain, temporal-anchor, and claim-policy checks.Source: §B.1 · p. 15

B.2 Release boundary and de-identification

  • The released HCC tables are de-identified while retaining random subject and visit identifiers for linkage, visit order, and longitudinal analysis.The release supports benchmark reproducibility, not clinical deployment.Source: §B.2 · p. 15

B.3 Functional groupings

  • Table 13 groups worksheets according to their roles in profiling and benchmark construction.Source: §B.3 · p. 15

B.4 Dataset limitations for benchmark design

Sparse detailed adverse-event records and absent billing or price fields constrain how benchmark safety and economics tasks are framed.

B.4 Dataset limitations for benchmark design

  • Sparse detailed AE/SAE records mean safety tasks distinguish screening-level summaries from detailed event-term analyses.Source: §B.4 · p. 15
  • Without direct billing or price fields, economics tasks use resource utilization, follow-up burden, or quality-of-life proxies rather than cost-effectiveness analyses.Source: §B.4 · p. 15

B.5 Benchmark task construction

The benchmark crosses four RWS objectives, three core questions per objective, and three prompt-specificity levels, using a shared output contract.

B.5 Benchmark task construction

  • The benchmark comprises 36 instances: four objectives crossed with three core questions and three prompt-specificity levels.Source: §B.5 · p. 16
  • Each objective–prompt cell averages three instances; each fixed prompt level contains twelve instances across the four objectives.The full benchmark includes all 36 instances across the three levels.Source: §B.5 · p. 16
  • Every instance uses the same output contract: a verification plan, executable analysis entry point, generated artifacts, and a structured RWS report.Source: §B.5 · p. 16

B.6 Public-RWS external validation suite

This section defines Public-RWS-18 as a separate external transfer suite and documents profile-grounded compilation, validation, repair, and certified reporting artifacts. It describes how profiles, typed study objects, evidence certificates, and auditable state artifacts constrain analyses, claims, and downstream decisions.

B.6 Public-RWS external validation suite

  • Public-RWS-18 tests transfer across public datasets and study designs, separately from the 36-instance HCC benchmark and without treating it as a full longitudinal RWS benchmark.Each dataset contributes two objectives, each evaluated with low-, medium-, and high-specificity prompts.Source: §B.6 · p. 16, Table 20 · p. 19, Table 21 · p. 20
  • Public-RWS-18 reuses the HCC profile construction, RWS-IR schema, validators, and claim checker, with dataset-specific claim policies as its only adaptation.Baselines without explicit evidence certificates receive post-hoc CTC, UCR, and CLV reconstruction; Scaffolded G3P gets templates, validator feedback, and the repair budget but not ProCARE’s compiler, certificate gate, reporting DAG, or planner state.Source: §B.6 · pp. 16, 19
  • The suite covers UCI Diabetes readmission and medication-utilization tasks, NHANES cross-sectional diabetes and medication-condition tasks, and TCGA-LIHC case-status and treatment/follow-up tasks.The prompts restrict conclusions to descriptive, proxy-association, or cross-sectional claims where appropriate, and avoid causal treatment-effect language.Source: §B.6 · pp. 19–21

B.7 Worksheet inventory

C.1 Profile schema Φ

  • Profiles are stored as four stable JSON blocks, with structural statistics and biomedical mappings consumed directly by downstream compilation, validation, and claim admission.Mappings undergo semantic-type, unit, value-range, and temporal-role checks; ambiguous or failed fields remain flagged and cannot authorize unsupported clinical roles or high-stakes claims absent an admitted proxy assumption.Source: §C.1 · p. 23

C.2 Concrete heterogeneous profile instance

  • A concrete HCC profile and recurrence-free-survival IR illustrate how sparse dates and unavailable fields constrain feasible analyses and support fallback reporting.The example certificate links a claim to its artifacts, code region, authorizing IR fields, profile entries, assumptions, and admission checks.Source: §C.2 · p. 23, §C.4 · pp. 25–26

C.3 RWS-IR schema and static checks

  • Static checks validate the typed RWS-IR’s study design, grain, joins, temporal anchors, variable roles, methods, artifacts, and claim policy before execution.Hard violations require repair, infeasibility, or downgrade; soft limitations can constrain claims, and only an IR with no remaining static violations enters execution.Source: §C.3 · p. 25

C.5 Execution validators and violation-guided repair

  • Execution-time validators check row-level temporal ordering, manifest completeness, figure creation, and artifact consistency, returning localized violations with repair guidance.Source: §C.5 · p. 26, Table 26 · p. 27

C.6 Evidence certificates and claim admission

  • ProCARE emits claim certificates during report generation, while the same checker reconstructs them post hoc for baselines and validates evidence links and policy-compliant wording.Each certificate links a supporting artifact, executed code region, authorizing RWS-IR field or method, relevant profile entry, and any recorded assumption or limitation.Source: §C.6 · p. 27
  • CTC, UCR, and CLV measure admitted evidence certificates, unsupported claims, and policy-exceeding language as proportions of extracted factual claims.Metrics are averaged per run across 12 tasks in each prompt-specificity group, with the overall row averaging all 36 HCC tasks; a claim may count as both unsupported and a language violation.Source: §C.6 · p. 28

C.7 Profile-grounded state artifacts

  • Profile-grounded state uses cached profiles and typed JSON artifacts, workspace files, skills, and validation logs rather than a vector database.Job artifacts can include RWS-IR, violation sets, execution specifications, code, logs, result and figure manifests, certificates, report sections, and disclosed assumptions; downstream stages reuse facts only when traceable to profiled tables, validated fields, executable artifacts, or recorded assumptions.Source: §C.7 · p. 28

C.8 Ontology mapping coverage

The section describes ontology mapping and the contracts that constrain ProCARE’s artifacts, execution, and reporting. Its recurrence-free survival example shows how evidence gates keep an unsupported safety analysis from being promoted as a result.

C.8 Ontology mapping coverage

  • Field mapping covers 0.83 of fields overall and 0.91 of critical task fields; unmapped fields are tagged for failures such as jargon, type incompatibility, ambiguity, or mixed units.The listed failure-category shares are 0.07, 0.05, 0.03, and 0.02, respectively.Source: §C.8 · pp. 28–29

C.9 Artifact bundle contract

  • Each run must return a validated RWS-IR, executable entry point, and evidence-certified report, while prompt specificity determines whether infeasible methods are stopped or disclosed as deviations.Under GOAL+METHOD, unavailable required data ends the analysis; GOAL and GOAL+APPROACH may yield the closest feasible analysis with limitations disclosed.Source: §C.9 · p. 29

C.10 Profile-constrained coding harness

  • The coding workspace checks schema, methodological, and artifact constraints through observable gates, requiring an explicit limitation or failed run when the profile cannot support a required variable, time anchor, or procedure.The workspace stores profiles, IR, code, logs, violations, and validation records, and rejects incomplete artifact bundles.Source: §C.10 · pp. 29–30

C.11 DAG reporting dependency rules

  • A dependency DAG orders report tasks using feasibility and support signals for variable availability, temporal validity, sample adequacy, and design consistency; traversal changes report ordering, not executed statistics.The fixed thresholds and weights prioritize validated support before promoting results over limitations or feasibility sections.Source: §C.11 · p. 30, Table 32 · p. 31

C.13 Standard report schema

  • The reporting schema defines required chapters, and report-section coverage is measured as the fraction containing non-trivial content.The minimal required chapters are listed in Appendix C.13.Source: §C.11 · p. 30, §C.13 · p. 31

C.12 Instantiated reporting DAG

  • In the recurrence-free survival instance, 1,215 of 1,295 surgery-cohort subjects had analyzable follow-up and 304 had valid recurrence events, while absent AE/SAE records blocked detailed safety profiling.The DAG routes the safety evidence gap to limitations rather than promoting it as a safety result.Source: §C.12 · p. 31, Figure 3 · p. 31

D Experimental Protocol and Reproducibility

The appendix specifies the execution and scoring setup, baseline adaptations, reproducibility materials, and the recurrence-free survival workflow artifacts. It also reports that scaffolded baselines narrow the execution gap, while ProCARE remains strongest on violation resolution and claim-level traceability.

D.1 Execution environment and cost accounting

  • The clean Python 3.9 sandbox disables network access, and reusable profiling lowers average token use from about 103k per query for one task to 39.8k across 36 tasks.The reported accounting uses approximately 65k profiling tokens and 38k inference tokens per query.Source: §D.1 · p. 31, T3 Endpoint and censoring T1,… · p. 32

T3 Endpoint and censoring T1, VT.PADHDAT, VT.PADRDAT, VT.VTDAT1, VT.VTDAT2

  • The recurrence-free survival task’s DAG specifies endpoint and censoring inputs, treatment exposure artifacts, feasibility checks, primary and sensitivity analyses, and a safety-evidence check.Its reporting artifacts include an RFS dataset, event/censor flags, temporal-validity log, and KM feasibility caveats.Source: T3 Endpoint and censoring T1,… · p. 32

D.2 Systems and baseline adaptation

  • Baseline comparisons give methods a shared prompt, data boundary, deliverables, and inference budget, while native code agents do not receive ProCARE’s profile, typed IR, certificate gate, or reporting DAG.A post-hoc contract checker assesses schema use, execution, artifact completeness, and claim traceability against profile-derived constraints.Source: §D.2 · p. 33

D.3 Fairness-controlled scaffolded baselines

  • Scaffolded controls add a schema profile, artifact template, sandbox, and repair budget to external baselines without providing ProCARE’s IR compiler, certificate gate, or reporting DAG.Methods lacking explicit certificates are scored using post-hoc evidence reconstruction.Source: §D.3 · pp. 33–34
  • Scaffolded baselines reduce N/A rates and narrow the execution gap, but integrated ProCARE remains strongest on violation resolution and claim-level traceability.The comparison is a fairness control, not a component ablation; all direct rows use Gemini-3-Pro.Source: Table 38 · p. 34, §D.3 · p. 34

D.4 Ablation protocol

  • Mechanism ablations remove one contract component at a time while holding the remaining defined workflow components fixed.The intervention definitions are specified in Table 39.Source: §D.4 · p. 35, Table 39 · p. 35

D.5 Model access, decoding, and version identifiers

  • Backbone sensitivity holds profiling, IR compilation, repair, certified reporting, prompts, and scoring fixed; the release includes code, prompts, de-identified benchmark tables, and evaluation scripts for reproduction.The release retains random benchmark IDs for reproducible linkage while removing direct identifiers.Source: §D.5 · p. 35, §D.6 · p. 35

D.6 Reproducibility package

  • The appendix provides local commands for reproducing runs and rebuilding the paper.The commands invoke the backend CLI, test suite, and appendix build.Source: §D.6 · p. 35

E Additional Results and Diagnostics

The appendix interprets claim and backbone diagnostics, then details the evaluation rubrics, review procedures, and prompts. It describes deterministic checks for artifact properties and blinded scoring for semantic quality.

E.1 Claim-diagnostic interpretation

  • More specific prompts increase claim traceability, while the dominant claim downgrades concern language policy, including causal phrasing from descriptive analyses and unsupported safety or subgroup claims.During certification, such claims are rewritten as limitations or removed from the admitted factual-claim set.Source: §E.1 · pp. 35–36

E.2 Backbone-sensitivity interpretation

  • Stronger backbones improve GA, PF, VRR, and CTC by using profile constraints and repair feedback more reliably, while deterministic validation and localized repair bound accepted artifacts after generation.Execution-only differences are smaller than planning and reporting differences.Source: §E.2 · p. 36

F System Interface and Workflow

G Evaluation Protocol and Rubrics

  • Evaluation combines deterministic feasibility and execution validators with LLM judges for semantic quality; validators check profile-valid IR fields, successful execution, required artifacts, temporal ordering, and chart-result consistency.For baselines, claim traceability is reconstructed from final artifacts; missing artifacts or unsupported variables block traceability.Source: G Evaluation Protocol and Rubrics · p. 37, §G.2 · pp. 37–38, §G.1 · p. 37

G.3 LLM judge protocol

  • Two blinded judges score anonymized artifact bundles with the same profile and evidence materials, and schema hallucination or artifact inconsistency causes hard rubric failures on affected dimensions.Scores are normalized and averaged across judges.Source: §G.3 · p. 38, §H.5 · p. 41

G.4 Expert-review protocol

  • Five blinded experts review the 36 primary HCC instances, with clinical and statistics/data reviewers assigned different stage responsibilities; stage scores average eligible reviewers’ 1–5 ratings.The appendix describes normalized calibration scores and correlations between automated and expert stage scores.Source: §G.4 · pp. 38–39, Table 44 · p. 40

G.5 Semantic Rubric Anchors

  • The semantic rubric defines judge scores for goal alignment, code quality, visual clarity, and scientific language, while the appendix identifies expert-score summaries, inter-rater reliability, and metric-correlation tables.The supplied passages describe these tables’ scope but do not give their result values.Source: §G.5 · p. 39, Table 45 · p. 40, Table 46 · p. 40

H.2 IR compiler prompt template

  • The prompt templates pass profile constraints into IR compilation, restrict coding repairs to profiled fields and methods, and require factual report sentences to carry certificates or be rewritten as limitations.The benchmark prompt also disallows hallucinated tables or columns.Source: §H.2 · p. 40, §H.1 · p. 40, §H.3 · p. 41

I Data Access, Privacy, and Responsible Use

The release supports research reproduction and auditing while restricting re-identification, participant linkage, and clinical use. Institutional deployment requires local review and revalidation, and the evaluation does not establish clinical truth, causal validity, regulatory sufficiency, or prospective safety.

Data access and responsible use

  • The HCC release removes direct identifiers but retains random subject and visit identifiers for reproducible joins; its terms prohibit re-identification, external linkage, and clinical use.Institutional deployment requires governance review, schema revalidation, and confirmation that local claim policies match available data.Source: I Data Access, Privacy, and… · p. 42

Evaluation scope and limitations

  • ProCARE is evaluated as a research automation and auditability system, not an autonomous clinical decision system, and its single-center, disease-specific HCC benchmark does not replace multi-institutional longitudinal validation.Porting to another institutional schema requires rebuilding the profile, revalidating IR checks, and confirming claim policies against the new data and study design.Source: I Data Access, Privacy, and… · p. 42
Loading 2610.16175v1…