PaperlayerView exact revision on arXiv

Source-linked AI summary

Dissecting Neuro-Symbolic Quality Assurance for Synthetic Oncology Data Generation

Laxmigayathri Challa, Yuhan Zhou, Ana Cleveland, Haihua Chen

arXiv:2608.22085v1cs.AI

TL;DR

Synthetic oncology data could address limited staging data, but clinically impossible hallucinations can contaminate downstream models, and the contributions of individual safeguards are unclear. The paper uses controlled ablations of generation-time gating, its constraints, and retrieval under fixed protocols, finding that schema validation does most filtering while retrieval effects vary by generator. Within this study, symbolic gating improves corpus validity without a commensurate gain on real lung-cancer notes.

  • Problem

    The study addresses uncertainty about which neuro-symbolic quality-assurance components protect synthetic oncology corpora from clinically harmful hallucinations.

  • Method

    Controlled ablations isolate the gate, its three constraints, and retrieval augmentation while holding generation and fine-tuning protocols constant.

  • Results

    Schema validation is the load-bearing filter: ungated generation admits 29.9% schema failures and 20.1% AJCC logic violations, while retrieval effects differ across generators.

  • Takeaways & Limitations

    Generation-time symbolic gating buys corpus validity, but ontology density should not serve as a proxy for quality and retrieval should be evaluated per model.

  • Takeaways & Limitations

    The gate does not constrain demographic plausibility or associations involving equity attributes, so these corpora should not be used to study disparities without additional auditing.

Abstract

from arXiv · show

Synthetic clinical data generation with large language models addresses the scarcity that limits cancer staging research, but oncology hallucinations are categorically harmful: one clinically impossible staging assignment contaminates every downstream model trained on it. Neuro-symbolic pipelines validate during generation, yet the contribution of individual quality-assurance components remains unclear. We report three controlled studies isolating gate necessity, constraint attribution, and retrieval conditionality, holding generation protocol, diversity thresholds, and fine-tuning hyperparameters constant across adapter conditions. The symbolic gate enforces schema completeness, ontology coverage against the Systematized Nomenclature of Medicine, and staging-logic consistency under American Joint Committee on Cancer eighth-edition rules. Ungated, 29.9% of records contain schema failures and 20.1% contain clinically invalid staging. Schema validation is the load-bearing filter: within the fully gated corpus it rejects 148 of 512 records, ontology grounding a further 24, and staging-logic validation none---the only generator producing logic violations is already excluded on schema, making clinical-logic validation a generator-conditional safeguard rather than the dominant filter. Retrieval augmentation is strongly model-dependent: it improves gate compliance for one generator by 12.5 percentage points, has no measurable effect for a second, and collapses output in a third. Across gated configurations ontology density is largely unchanged, indicating that symbolic validation improves clinical validity rather than vocabulary richness. Symbolic gating therefore buys corpus validity but no commensurate gain on real lung-cancer notes in this study; retrieval should be evaluated per model, and ontology density should not be reported as a proxy for corpus quality.

I. INTRODUCTION

This study decomposes a neuro-symbolic synthetic oncology-data pipeline to identify which quality-assurance components protect corpus validity and downstream utility. Controlled experiments isolate gating, individual constraints, and retrieval while holding generation and fine-tuning conditions fixed.

  • Research questions: The study asks what ungated generation admits, which symbolic constraint contributes most, and when retrieval augmentation improves generation quality.These questions are tested through controlled component manipulations.
  • Study design: Five matched adapter conditions isolate the gate, its three constraints, and retrieval as independently testable variables.Generation and fine-tuning protocols remain fixed across conditions.
  • Why gating matters: Ungated corpora admit failures that are structurally indistinguishable from valid records, so post-hoc inspection cannot recover the contamination.The motivation is especially acute for clinically consequential oncology hallucinations.
  • Component attribution: Clinical-logic validation is a generator-conditional safeguard rather than a high-volume filter, whereas the study ranks symbolic constraints by marginal contribution.The decomposition separates filtering work from standardization work.
  • Retrieval conditionality: Gate compliance serves as a per-generator diagnostic for whether retrieval will help, do nothing, or cause output collapse.This frames retrieval as a conditional design choice rather than a universal improvement.

II. RELATED WORK

Prior clinical synthetic-data and hallucination research generally validates records after generation, while this work treats ontology coverage and clinical logic as generation-time admission constraints. It also tests whether retrieval universally improves structured clinical generation, finding that its effect depends on the generator.

  • Post-generation validation: Earlier GAN, mechanistic, and LLM systems produce records first and apply quality checks or ontology mapping afterward.This post-generation pattern limits validation to records that already exist.
  • Generation-time gating: This work enforces ontology coverage and clinical-logic consistency as record-level admission constraints during generation.The approach differs from normalization and token-level decoding constraints.
  • Hallucination prevention: Clinical hallucination benchmarks and clinician-in-the-loop analyses primarily measure hallucination after generation, whereas AJCC rules enable binary validation of clinically impossible staging.The gate converts categorical failures into an auditable engineering property.
  • Filter versus gate: The architectural distinction between filtering and gating makes admission independently measurable while avoiding the cost of retaining records that fail constraints afterward.Varying active constraints under fixed conditions supports component attribution.
  • Retrieval augmentation: Retrieval improves one generator, has no measurable effect on a second, and causes systematic output collapse in a third.The result challenges the assumption that retrieval universally improves structured clinical generation.

E. TSTR Evaluation and TNM Staging Automation

The paper evaluates synthetic oncology notes with train-on-synthetic, test-on-real staging extraction and grounds quality assessment in deterministic TNM logic and SNOMED CT coverage. It emphasizes real-world staging difficulty and cautions against misleading metrics or vocabulary proxies.

  • TSTR evaluation: Train-on-Synthetic, Test-on-Real measures synthetic-data utility through downstream performance on TCGA-derived lung-cancer and cross-tumor clinical note cohorts.The evaluation uses explicit staging annotations.
  • Staging automation: Real-world oncology-note performance remains substantially below curated-benchmark performance because clinical dictation is noisy, abbreviated, and inconsistently formatted.This motivates synthetic data targeted at structured oncology staging.
  • TNM staging: TNM values determine AJCC 8th Edition stage groups through a fixed lookup, making staging violations logically impossible rather than probabilistically approximate.T, N, and M encode tumor extent, regional nodes, and distant metastasis.
  • Clinical-logic validation: Staging contradictions can pass structural formatting, ontology-density, and JSON-schema checks, so only an explicit clinical-logic validator detects them.Examples include M1 without Stage IV or Stage I with N2 involvement.
  • Ontology assessment: SNOMED CT coverage supplies a validity anchor, while length-normalized density measures vocabulary richness rather than directly establishing overall corpus quality.The analysis tests whether ontology admission constraints enrich vocabulary.

IV. METHODOLOGY

The methodology couples LLM generation with a per-record symbolic gate and controlled ablations that isolate gate, constraint, and retrieval effects while holding other settings fixed.

  • Pipeline: The pipeline applies G(x) = S ∧ O ∧ C before admitting each generated record to the training corpus.S checks rigid.v3 schema completeness, O confirms SNOMED CT coverage, and C enforces selected AJCC staging rules.
  • Evaluation: Pipeline quality is assessed using schema compliance, TNM label diversity, and downstream TSTR utility.Diversity uses Shannon entropy floors, while utility measures T-, N-, and M-stage extraction accuracy and macro-F1.
  • Experimental design: Five matched adapter conditions manipulate one pipeline component at a time while generation, diversity, hyperparameters, and evaluation remain fixed.The conditions span ungated, schema-only, schema-plus-ontology, full-gate no-retrieval, and full-gate retrieval settings.
  • RQ1: Gate necessity: The gate necessity contrast compares ungated Adapter A with full-gate Adapter D.Both corpora use the same TNM grid and entropy floors, differing in whether all symbolic constraints control admission.
  • RQ2–RQ3: Constraint attribution compares schema-only, schema-plus-ontology, and full-gate conditions, while retrieval conditionality compares full-gate no-RAG with full-gate RAG.These contrasts correspond to B/C/D and D/E, respectively.

C. Ablation Design

The gate-necessity ablation compares matched ungated and fully gated corpora, using the same generation setup and evaluating five adapter conditions within a common design.

  • Gate necessity: Adapter A admits every JSON-parsing record, whereas Adapter D admits only records passing G(x) = S ∧ O ∧ C.Both corpora use the same TNM grid, models, hyperparameters, and entropy floors.
  • Retrieval implementation: Retrieval fallback occurs for 64 of 448 calls when the MedCPT index returns no qualifying match.These calls involve Llama-3.3-70B and ClinicalCamel-70B; all GPT-4o calls use MedCPT.
  • Matched conditions: Figure 2 defines five matched conditions: ungated, schema-only, schema-plus-ontology, full-gate no-RAG, and full-gate RAG.The design maps A versus D to gate necessity, B/C/D to constraint attribution, and D versus E to retrieval conditionality.
  • Ablation mapping: Adapter D is the shared full-gate, no-RAG baseline across all three ablations.No-RAG conditions total 512 generations, while RAG conditions total 448; all adapters use identical QLoRA settings.

2) Ablation 2: Constraint Attribution (RQ2):

Constraint attribution progressively enforces schema, ontology, and clinical logic, then compares matched full-gate retrieval conditions across standardized TSTR evaluations.

  • Constraint attribution: The three progressively stricter corpora enforce schema only, schema plus ontology, and the full schema-plus-ontology-plus-logic gate.Adapters B, C, and D correspond to these conditions.
  • Attribution method: Marginal constraint contributions are computed from per-record S, O, and C flags within one corpus rather than independent corpus samples.This design reduces attribution to the constraints themselves rather than generation-to-generation sampling variation.
  • Retrieval conditionality: The retrieval comparison uses matched full-gate corpora that differ only by prepended MedCPT context.Both use the same TNM grid, three generators, gate, and entropy floors.
  • Evaluation protocol: The adapters are evaluated under identical TSTR conditions on synthetic held-out data, TCGA Lung notes, and TCGA Cross-Tumor notes.These represent in-distribution, real lung-cancer, and out-of-distribution oncology evaluations.
  • Results: 29.9% of ungated records contain schema failures, while 20.1% contain AJCC logic violations.The full gate rejects one record in three, with rejection concentrated in one generator.
  • Results: 148 of 512 records are rejected by schema validation, 24 more by ontology grounding, and none by clinical-logic validation.Clinical-logic validation is therefore generator-conditional rather than the dominant high-volume filter in this corpus.
  • Results: Retrieval changes gate compliance by +12.5 percentage points for one generator, has no measurable effect for another, and collapses output for a third.The result supports evaluating retrieval separately for each generator.
  • Results: Across gated configurations, SNOMED density remains nearly flat at 29.71–29.86 terms per 100 words.The reported corpus-quality gains do not yield commensurate improvement on real lung-cancer notes.

A. Corpus Quality Across Ablation Conditions

Across controlled ablations, symbolic gating primarily removes structurally invalid records, with schema validation contributing the largest marginal filter while ontology and logic checks contribute less under these generators.

  • Gate necessity: 29.9% of ungated records failed schema validation, while 20.1% violated AJCC clinical logic.These failures would enter fine-tuning without the gate.
  • Model dependence: 100% of GPT-4o records passed the full gate, while Llama’s full-gate yield was 87.5% and ClinicalCamel’s schema compliance remained 22.9%.The gate cost therefore varied sharply by generator model.
  • Constraint attribution: 148 of 512 records were rejected by schema validation, 24 by ontology grounding, and 0 by AJCC logic validation.Schema rejected all marginal failures from ClinicalCamel, whereas ontology rejected 24 Llama records.
  • Vocabulary versus validity: 29.71, 29.86, and 29.83 SNOMED terms per 100 words were observed across progressively stricter gate levels.The near-flat densities indicate validity filtering rather than vocabulary enrichment.
  • Constraint attribution: ClinicalCamel’s logic-violating records also failed schema, so logic validation produced no marginal rejections in this study.The logic gate remains conditional on a generator producing schema-valid but clinically impossible records.

D. RQ3 (Retrieval Conditionality) — Retrieval-Augmented vs. Non-Augmented Generation

Retrieval augmentation had sharply model-dependent effects: it improved Llama’s gate compliance, added no measurable signal for GPT-4o, and caused substantial output collapse for ClinicalCamel.

  • Aggregate effect: 69.2% of RAG records passed the gate versus 66.4% without RAG, a +2.8 pp aggregate increase.Admitted-record SNOMED density and unique concept coverage were essentially unchanged.
  • GPT-4o: GPT-4o’s per-run density delta was −0.20 terms per 100 words with p = 0.498, while gate pass rate remained 100% in both conditions.Retrieval added no measurable signal for this already-compliant model.
  • Llama-3.3-70B: Llama-3.3-70B gate pass rate increased from 87.5% without RAG to 100.0% with RAG.Retrieval resolved ontology failures without enriching the density of already-admitted records.
  • ClinicalCamel-70B: Under RAG, 75 of 160 ClinicalCamel runs produced exactly 0.0 SNOMED terms per 100 words, and gate pass rate fell from 22.9% to 13.8%.The collapse began around run 85 and caused 138 of 160 runs to fail the gate.

E. Downstream Utility: Train-on-Synthetic, Test-on-Real Evaluation

Synthetic-trained adapters performed well on held-out synthetic data but did not exceed the majority-class baseline on TCGA Lung; transfer appeared only on the more heterogeneous Cross-Tumor benchmark.

  • In-distribution evaluation: Across the held-out synthetic benchmark, all adapter conditions maintained consistently high accuracy and macro-F1.Differences between adapters were modest in-distribution.
  • TCGA Lung: No adapter exceeded the majority-class baseline on any axis of the TCGA Lung pathology benchmark.Cleaner synthetic corpora did not yield commensurate lung-cancer generalization in this evaluation.
  • TCGA Cross-Tumor: On TCGA Cross-Tumor, every adapter cleared the T-stage baseline, scoring 0.59–0.62 against a baseline of 0.35.Adapters C and D also recorded the strongest N-stage performance.
  • Metric interpretation: M-stage accuracy remained higher than macro-F1 across conditions because the evaluation sets were strongly imbalanced toward M0 cases.M-stage accuracy should therefore not be interpreted as superior metastatic classification performance.
  • Overall interpretation: The study ranks schema validation as load-bearing, ontology grounding as a smaller filter, and logic validation as generator-conditional, without showing commensurate lung-note transfer.The downstream evaluation is part of the component-attribution analysis.

B. The Gate Filters Validity, Not Vocabulary

Gate strictness did not materially increase SNOMED density, separating vocabulary recognition from clinical correctness and motivating per-generator retrieval decisions.

  • Gate effects: 29.71, 29.86, and 29.83 SNOMED terms per 100 words were measured across gate levels B, C, and D.The near-flat values falsify vocabulary enrichment as the gate’s primary effect.
  • Quality metrics: A record can contain 35 SNOMED terms per 100 words while assigning M1 without Stage IV, making it ontologically rich but clinically invalid.Vocabulary density and clinical correctness are independently evaluable properties.
  • Constraint roles: The ontology constraint standardizes vocabulary, whereas the clinical-logic constraint is the mechanism intended to catch logical staging errors.Their contributions are qualitatively different rather than redundant.
  • Retrieval conditionality: Llama’s RAG gate pass rate rose from 87.5% to 100.0% by resolving ontology failures rather than enriching already-admitted records.Retrieval therefore acted as a compliance mechanism for this generator.
  • Retrieval conditionality: GPT-4o gained nothing measurable from retrieval because its gate was already satisfied at 100%.For ClinicalCamel, retrieval instead caused systematic output disintegration.

E. Toward Downstream Transferability

The study finds that symbolic gating secures corpus validity, but downstream transfer remains limited by narrow synthetic language and the synthetic-to-real gap. It also identifies scope boundaries around yield, single-run comparisons, and unconstrained demographic plausibility.

  • Limitations: The fixed 32-cell TNM grid and templated prompts constrain lexical and structural variety, so a clinically clean corpus transfers imperfectly to real pathology narratives.
  • Findings: Schema validation does most of the filtering, while clinical-logic validation removes no additional records in this study.The only logic-violating generator fails schema first, making clinical-logic validation a generator-conditional safeguard rather than a high-volume filter.
  • Findings: Retrieval augmentation is model-conditional, resolving ontology failures for one generator, adding nothing for a second, and collapsing output for a third.
  • Design implication: Generation-time admission constraints make hallucination detection binary, auditable, and reproducible, but reduce corpus yield.
  • Downstream transferability: Gated and ungated adapters were largely indistinguishable on T-stage across real TCGA pathology reports.The study locates remaining downstream leverage in broader, more varied, longitudinal synthetic records rather than stricter filtering alone.
  • Limitations: The gate does not constrain demographic associations with stage, histology, or treatment intent, and such corpora should not be used to study disparities without an audit.

APPENDIX A ENTROPY FLOOR DERIVATION AND LABEL DIVERSITY

The appendix derives entropy floors from theoretical label maxima and documents the fixed prompts and training configuration used across adapters. Balanced TNM seeding keeps admitted-corpus diversity at the maxima and above the floors.

  • Entropy floor derivation: 80% entropy floors equal 1.109 nats for T and N and 0.554 nats for M.The maxima are ln 4 = 1.386 nats for T and N, and ln 2 = 0.693 nats for M.
  • Entropy floor derivation: No entropy intervention was triggered at either checkpoint in any of the five adapters.Floors are checked after generation on the admitted corpus and before training on the stratified training partition.
  • Label diversity: All conditions reach the theoretical maxima because the 32-cell seed grid balances labels by construction.
  • Evaluation: Table VII reports per-axis macro-F1 for five adapters under train-on-synthetic, test-on-real evaluation across three cohorts.Macro-F1 weights classes equally and is especially informative for minority-class recovery in M-stage.
  • Training controls: All five adapters use identical QLoRA hyperparameters and the same training seed, but one seed does not bound seed-to-seed variance.
  • Prompt families: The structured prompt conditions generation on one TNM seed and requires a complete rigid.v3 JSON record, while the extraction prompt returns T, N, and M keys.The RAG condition prepends top-k MedCPT-retrieved PubMed abstracts to the structured-generation template.
Loading 2608.22085v1…