Answering clinicians' questions over trial evidence tables with verifiable, feedback-driven language models

• Evidence tables are central products of systematic reviews, but clinicians often need an analyst to filter them for new questions.Source: §1 · p. 2
• Some questions require attributes absent from the table, such as target class or a harmonised endpoint, which reviewers re-derive inconsistently.Source: §1 · p. 2
• Prior tools leave three gaps: cohort selection errors, unrecorded attributes, and expert corrections that are not reused.Source: §1 · p. 2
• FD-SCoPE routes recorded-field questions to read-only SQL and unrecorded-attribute questions to trial selection plus checked derivation, recording steps for inspection.Source: §1 · p. 2, Figure 1 · p. 3
• FD-SCoPE is presented as answering clinician-style evidence tasks reliably, finding trials and deriving unrecorded attributes better than four alternatives, and improving after corrections, including on uncorrected trials.Source: §1 · pp. 2–3

2.2 FD-SCoPE completes clinician-style evidence tasks

• 100% of 140 clinician-style tasks were answered correctly, compared with 90.7–97.9% for seven alternatives.The alternatives’ errors mostly involved names differing from stored values or conditions assigned to the wrong field; FD-SCoPE resolved names against table values before querying.Source: §2.2 · p. 4, Figure 2 · p. 5
• 58.9% exact-match accuracy on 1,500 recorded-field benchmark questions was highest among eight systems, 3.7 points above the DAIL-SQL-style baseline.Every FD-SCoPE query was executable; execution F1 was 67.4%, versus 67.1% for the DAIL-SQL-style baseline.Source: §2.2 · p. 4, Figure 2 · p. 5

2.3 Finding the right trials and deriving unrecorded attributes

• 77.7% derived-value F1, versus 73.4% for the BlendSQL-style baseline and 64.8–67.9% for three other approaches; FD-SCoPE retrieved 99.3% of relevant records at 89.8% positive predictive value.It returned the exact reference trial set for 84.3% of questions.Source: §2.3 · pp. 4–5, Figure 3 · p. 6

2.4 Expert corrections become reusable rules

• After simulated corrections on 299 questions, held-out derived-value F1 rose from 77.9% to 84.9%; exact whole-answer agreement rose from 52.6% to 68.5%.Trial selection was unchanged, so the reported gain came from derivation; individual-value accuracy also rose from 84.6% to 92.6%.Source: §2.4 · p. 6, Figure 4 · p. 7
• Feedback gains were largest for dosing schedule, composite-endpoint components, primary endpoint type, and platinum agent, while anti-VEGF F1 fell from 100% to 5.9%.Ten of 31 derivation families improved by more than one point, and learned rules remained subject to expert approval.Source: §2.4 · pp. 6–7, Figure 5 · p. 8

2.5 Query-based trial selection, planning and approved examples account for the gains

• Replacing query-based trial selection with model selection lowered derived-value F1 from 78.0% to 67.6%; removing the planner lowered F1 to 72.0%.For recorded-field questions, removing approved-example retrieval lowered exact match from 62.0% to 56.0%, while the full example bank reached 62.0%.Source: §2.5 · p. 7, Figure 6 · p. 9

2.6 Robust to clinical phrasing and read-only by design

• Rewording questions with case, brand names, disease abbreviations, or drug-class shorthand changed exact match by −0.3 to +1.3 points, while clinical-term typos cost 5.9 points.Execution F1 changed by similar amounts.Source: §2.6 · p. 8, Figure 6 · p. 9
• The scope check declined all 36 out-of-scope prompts and answered all 10 valid controls; with that check off, the query guard still made no data-modifying statement.The table file was byte-identical before and after the test.Source: §2.6 · p. 9, Figure 6 · p. 9

3 Discussion

• On the 159-record oncology table, FD-SCoPE completed every clinician-style task, found nearly all relevant trials for unrecorded-attribute questions, and improved derived-value accuracy over four alternatives.The conclusion also reports improvement after corrections to a fifth of the questions.Source: §3 · p. 9
• Query-based selection lets reviewers distinguish trial-selection errors from derivation errors, and replacing it with model-selected rows lowered accuracy by ten points.The authors regard silently missing trials as more consequential than a wrongly derived label visible beside its source value.Source: §3 · pp. 9–10
• Validated rules can transfer corrections to later questions and unseen trials, but applying a rule outside its learned family can produce confidently wrong answers.The authors therefore regard expert approval, with the validation record visible, as required.Source: §3 · p. 10
• FD-SCoPE complements systems that build evidence tables; on recorded-field questions, its advantage over the strongest text-to-SQL baseline was modest and it used about twice as many tokens per question.Its larger differences were on derived attributes and tasks involving named trials and drugs.Source: §3 · p. 10
• Evaluation used one oncology table of 159 records, no clinician users, author-generated clinician-style tasks, simulated feedback, and a single run per system.The study therefore does not establish the expert effort required for feedback or run-to-run variation.Source: §3 · p. 10
• The authors propose prospective evaluation with reviewers and describe FD-SCoPE as an interface to already curated evidence, not a decision-making system.They identify multi-table registries, ontology-based name normalisation, and calibrated abstention as technical priorities.Source: §3 · p. 11

4.1 Study design and evaluation framework

• The evaluation measures trial identification, evidence-table accuracy and derived-attribute accuracy across direct-retrieval and derivation questions.Derivation applies a clinical or analytical rule to recorded information, such as mapping nivolumab to PD-1.Source: §4.1 · p. 11

4.2 Clinical-trial evidence source

• The source is a read-only IOTOX SQLite table containing 159 oncology trial records and 32 fields.Its stored terminology, including abbreviations and variant spellings, was retained as encountered in structured evidence.Source: §4.2 · pp. 11–12

4.4 Question sets and data partitions

• The benchmarks include 1,500 recorded-field questions, 1,500 derived-attribute questions across 31 families, and 140 clinician-style tasks.The derived set hides fields that directly reveal target values; the clinician-style tasks were generated after system development.Source: §4.4 · p. 12

4.5 FD-SCoPE architecture

• A scope gate and router distinguish recorded-field requests from derivation requests, with benchmark pathways fixed to avoid conflating routing errors with accuracy.The gate declines requests to modify data, ignore system rules, provide clinical advice, or return patient-level data.Source: §4.5 · pp. 12–13

4.5.1 Clinical concept grounding and exemplar retrieval

• The system decomposes each question into up to six grounded constraints and retrieves similar clinician-approved examples as structural guidance.A catalogue of schema fields and observed values supports exact, normalized, alias-based, and approximate matching; approximate matching is disabled for identifiers and trial names.Source: §4.5.1 · p. 13

4.5.2 SQL generation and guarded execution

• For recorded fields, FD-SCoPE generates one read-only SQLite query and checks it for valid fields, terminology, and non-modification before execution.A failed valid query can receive one revision, while empty results are not automatically broadened.Source: §4.5.2 · p. 13

4.5.3 Hybrid retrieval and derivation

• For unrecorded attributes, SQL first selects trial records, then a planner specifies the source field and derivation rule for those records.A stored or validated program is preferred; otherwise the model derives values row by row, with program execution sandboxed and failures reverting to row-level derivation.Source: §4.5.3 · pp. 13–14

4.6 Clinician feedback and feedback-driven learning

• Experts can approve answer tables or correct values and derivation rules, which update exemplar and program banks under agreement and validation thresholds.For programs with at least six distinct available values, 30% are held out for validation; look-up families use a 95% agreement threshold when unseen-value validation is unavailable.Source: §4.6 · p. 14
• The controlled feedback experiment supplied reference answers for 299 feedback questions and evaluated on 1,201 held-out questions, with banks frozen before testing.A second condition restricted program learning to half of registry identifiers and assessed values from the other half.Source: §4.6 · p. 14

4.7 Prompt design

• Stage-specific prompts constrain component outputs to machine-readable formats; reported experiments did not exercise routing or back-translation prompts.No query required model repair, and fallback row selection was used for four derived-attribute questions and in component analysis.Source: §4.7 · pp. 14–15

4.8 Alternative approaches

• Comparisons included prompting and adapted text-to-SQL baselines for recorded fields, plus distinct retrieval, code-agent, and whole-table baselines for derived attributes.All systems shared the model, table, read-only database, and evaluation procedure, but comparison systems lacked FD-SCoPE’s decomposition outputs, exemplar bank, output-field completion, or guarded repair.Source: §4.8 · p. 15

4.9 Component analysis, robustness and safety

• Component analysis removes individual derivation or retrieval components while holding other settings fixed and resetting learning state for each run.Derived-attribute variants remove program induction, the planner, normalization, or query-based selection; recorded-field variants remove grounding, retrieval, field completion, or decomposition.Source: §4.9 · pp. 15–16
• Robustness testing applies meaning-preserving rewrites, including lower case, brand names, disease abbreviations, class shorthand, and clinically relevant typos.Only questions to which a rewrite applied were included, with the reference query unchanged.Source: §4.9 · p. 16
• The safety set contains 46 prompts and compares runs with the scope gate enabled versus disabled while keeping the execution guard enabled.The set covers database modification, prompt injection, clinical advice, patient-level information, and appropriate read-only controls; database hashes were compared before and after each run.Source: §4.9 · p. 16

4.10 Outcome measures

• Execution exact match requires a predicted query to reproduce the reference result, while execution F1 summarizes row-level precision and recall after field alignment.Additional context fields are permitted when all expected fields and values are present.Source: §4.10 · p. 16
• Derived-value F1 and exact form are computed after aligning rows on trial identifiers, publication identifiers, trial name, and source value.The measures normalize case, whitespace, and equivalent label forms; question-level all-values-correct and whole-table measures are also reported.Source: §4.10 · p. 16
• Trial-level sensitivity and positive predictive value compare retrieved trial records with the reference set, and exact trial set measures the share of questions with identical sets.Empty answers count as correct for questions whose reference table is empty.Source: §4.10 · p. 16

4.11 Statistical analysis

• Measures were computed for each question and averaged.The passage specifies no further aggregation detail.Source: §4.11 · p. 16
• 95% percentile intervals use 1,000 non-parametric question-level bootstrap resamples, while clinician-style tasks use Wilson intervals.Paired system differences use 10,000 bootstrap resamples; no hypothesis tests were performed.Source: §4.11 · p. 16
• Each system ran once, so intervals reflect variability across questions rather than repeated-run variability.Source: §4.11 · p. 16

4.12 Models and computational environment

• Qwen3.8-27B with thinking mode disabled was the primary model, and Gemma-3-27B-it was used for model comparison.Source: §4.12 · p. 16
• Models were served through an OpenAI-compatible interface on NVIDIA H200 GPUs in bfloat16, with decoding temperature 0.0 and top-p 1.0; the C3 style baseline alone differed in sampling step.Source: §4.12 · p. 17
• Served context limits were 40,000 tokens for Qwen3.8-27B and 24,000 for Gemma-3-27B-it.Orchestration, database execution, sentence embedding, and evaluation ran on CPUs under Python 3.12.Source: §4.12 · p. 17

4.13 Reproducibility and software verification

• Runs retained resolved configurations, predictions, token summaries, and evaluation reports; FD-SCoPE also stored structured traces of pathways, queries, executions, and derivations.The software passed 21 deterministic offline tests covering components from grounding and routing to end-to-end evaluation.Source: §4.13 · p. 17

Data availability.

• The IOTOX living evidence table, question sets, clinician-style tasks, and evaluated-system outputs are available.Source: Data availability · p. 17

Code availability.

• The source code, prompts, run configurations, evaluation procedures, and figure and supplement-generation scripts are available.Source: Code availability · p. 17

Supplementary information.

• Supplementary Tables S1–S14 and the complete model prompts in the Supplementary Methods follow.Source: Supplementary information · p. 17

Supplementary Tables

• Table S1 reports exact match, execution F1, and token use for the 1,500 recorded-field benchmark questions.Exact match requires reproducing the reference table; execution F1 measures returned-record agreement.Source: Table S1 · p. 18, Supplementary Tables · p. 18
• On recorded-field questions, trial-set scoring uses sensitivity and positive predictive value, and systems were within a few points of one another.Source: Table S2 · p. 18, Supplementary Tables · p. 18
• Tables S3 and S4 report derived-value accuracy and trial retrieval before derivation, including exact trial-set matches.Table S3 includes normalized and exact derived-value F1 measures.Source: Table S3 · p. 19, Supplementary Tables · p. 19, Table S4 · p. 19
• Supplementary Table S5 reports feedback results on held-out questions and repeats the experiment using corrections from half the trials.Corrections were simulated from reference answers to feedback questions.Source: Table S5 · p. 20, Supplementary Tables · p. 20
• Tables S6, S10, and S11 report derivation-family F1, reworded-question accuracy, and paired system differences.Source: Table S6 · p. 21, Supplementary Tables · pp. 21, 23, Table S10 · p. 23
• Tables S7 and S8 report component analyses for recorded-field questions and derived-attribute questions, respectively.Table S7 also examines partial approved-example banks; Table S8 includes a variant where the model selects rows from the whole table.Source: Table S7 · p. 22, Supplementary Tables · p. 22, Table S8 · p. 22
• Table S9 reports correct-task counts overall and across six clinician-style task types, with Wilson intervals for the share of 140 tasks.The task types include named-trial lookup, publication counts, cancer-type landscapes, enrolment and recency, non-chemotherapy controls, and PD-L1-positive eligibility.Source: Table S9 · p. 22, Supplementary Tables · p. 22
• Table S12 reports safety-prompt outcomes with the scope check on and off; without it, the query guard is the only protection.The safety prompts were evaluated as read-only answers, and the table file was byte-identical before and after both runs.Source: Table S12 · p. 24, Supplementary Tables · p. 24

Supplementary Methods: complete model prompts

• Runtime prompt values include the clinical question, schema, observed values, grounded predicates, exemplars, selected rows, and intermediate outputs.Source: Supplementary Methods: complete model… · p. 25

Question routing

• Routing distinguishes stored-field questions for SQL from latent-value questions that require per-row derivation.Source: Question routing · pp. 25–26

Question decomposition

• Decomposition turns each constraint into an atomic retrieval question mapped to one SQL WHERE predicate, using canonical columns, values, and explicit operators.Source: Question decomposition · p. 26

SQL synthesis and SQL repair

• SQL synthesis produces one read-only SELECT using the question, grounded predicates, and exemplars, while repair changes as little as possible without altering constraints.Source: SQL synthesis · p. 27, SQL repair · p. 28

Fallback row selection

• Fallback row selection uses visible table rows to identify matching records and returns their row IDs without deriving the final answer.Source: Fallback row selection · pp. 28–29

Hybrid derivation planning and Row-level derivation

• Hybrid planning specifies source columns, derivation and normalization rules, and an optional deterministic Python expression; row-level derivation applies the plan to each candidate row.Source: Hybrid derivation planning · p. 29, Row-level derivation · p. 30

SQL back-translation and Derivation-program induction

• SQL back-translation preserves filters and thresholds when rewriting a query as a clinician’s question, while program induction learns a Python expression from clinician-approved examples.Source: SQL back-translation · p. 31, Derivation-program induction · pp. 31–32

Comparator prompts; Zero-shot prompting; Few-shot prompting; Chain-of-thought prompting

• Comparator prompts include zero-shot schema-based SQL generation, four fixed clinician-approved few-shot demonstrations, and a chain-of-thought scaffold for normalizing terms and mapping constraints.Source: Zero-shot prompting · p. 32, Few-shot prompting · p. 32, Chain-of-thought prompting · p. 33