Source-linked AI summary
Adapt Semantics, Not Structure: Few-Instance Schema Calibration for Scientific PDF Extraction
Zixiao Dong, Wei Yang, Zihao Liu, Chenshu Li, Longzhang Liu, Tao Tan, Hong Xie
TL;DR
Scientific PDF extraction schemas can remain difficult for LLMs to execute reliably, and verified examples are costly to obtain. CPSE calibrates prompts and field semantics while preserving the output structure and separating identity discovery from record completion. On expert-annotated polymer-science PDFs, it gains +9.93 over an execution-matched baseline, with gains also reported under an independent judge and blinded expert audit.
Problem
Scientific PDFs distribute evidence across text, tables, and figures, while field-level extraction semantics are often unspecified and verified annotations are costly to obtain.
Source: §1 · p. 1Method
CPSE jointly calibrates prompts and field-level descriptions while freezing the structural contract, and separates identity discovery from record completion using a material manifest.
Source: §1 · p. 1Results
Against the execution-matched manifest-conditioned baseline of 81.02, CPSE gains +9.93; gains also appear under an independent judge and in a blinded expert audit.
Source: §4.3 · p. 4Takeaways & Limitations
A few verified examples can improve schema execution on expert-annotated polymer-science PDFs without altering the downstream output contract.
Source: §5 · p. 4
Abstract
from arXiv · showhide
A well-designed extraction schema is not necessarily ready for reliable LLM execution. When only limited verified extractions are available, manually tuning hundreds of field definitions through trial and error is costly. We frame this problem as few-instance schema calibration: adapting the operational semantics of an existing schema from a few annotated documents while preserving its structural contract. We introduce CPSE, a contract-preserving semantic extraction framework that jointly calibrates extraction prompts and field-level semantic descriptions from a few gold annotations. CPSE decomposes the schema into an invariant structural contract and mutable field semantics, and further separates identity discovery from record completion using manifest-conditioned resolution. On expert-annotated polymer-science documents, CPSE improves extraction by 9.93 points over an execution-matched baseline, with consistent gains under an independent judge and in a blinded expert audit. These results show that CPSE enables low-resource schema execution while preserving the output structure required downstream.
1. INTRODUCTION
Scientific PDF extraction requires both an adequate schema and reliable field-level interpretation, but verified annotations are costly. CPSE calibrates prompts and field semantics from a few examples while preserving the schema structure and separating identity discovery from record completion.
1. INTRODUCTION
- Scientific PDFs distribute evidence across text, tables, and figures, while field interpretation depends on experimental context.Reliable extraction therefore requires both an adequate representational schema and faithful LLM execution.Source: §1 · p. 1
- Limited verified annotations make manual tuning costly, especially for schemas with hundreds of fields; global instructions leave field-level interpretation unresolved, while schema changes risk the downstream interface.These limitations motivate adapting field semantics while keeping the structure fixed.Source: §1 · p. 1
- CPSE targets post-design calibration: refining an established schema’s operational semantics under limited supervision without changing its structural contract.The contract includes field paths, types, cardinalities, requiredness, and nesting; field descriptions can distinguish similar quantities without changing paths or types.Source: §1 · p. 1
- Single-pass extraction couples material-identity discovery with record completion, making record boundaries difficult to maintain in entity-dense documents.CPSE addresses this by separating identity discovery from full-record generation with manifest-conditioned resolution.Source: §1 · p. 1
- CPSE jointly calibrates extraction prompts and field-level semantic descriptions through PDF-aware textual feedback while holding the schema structure fixed.Its source-ordered material manifest makes record boundaries explicit; experiments report gains across ablations, external optimizers, independent evaluation, and a blinded expert audit.Source: §1 · pp. 1–2
2. RELATED WORK
Related work addresses scientific information extraction, prompt optimization, mutable schema representations, and document-level entity association. CPSE instead targets full-PDF extraction into a fixed nested JSON contract while calibrating field semantics and making sample boundaries explicit.
2. RELATED WORK
- Prior scientific and polymer-specific systems extract entities, measurements, and material–property records, whereas CPSE targets full-PDF extraction into a fixed nested JSON contract from a few annotated documents.The cited polymer systems use abstracts or selected full-text content.Source: §2 · p. 2
- Prompt optimizers tune instructions or textual program components, while schema-oriented methods modify schema representations; CPSE keeps output structure invariant and calibrates prompts with field-level semantics.This targets field-level operational semantics while retaining the downstream interface.Source: §2 · p. 2
- Document-level extraction methods associate distributed evidence with entities; CPSE first builds a source-ordered material manifest, then resolves small, non-overlapping identity subsets separately.This makes sample boundaries explicit and limits cross-sample mixing.Source: §2 · p. 2
- Unlike grammar-constrained decoding, which enforces structural validity, CPSE calibrates field semantics and evidence assignment under a fixed structure.Source: §2 · p. 2
3. METHOD
CPSE begins with an induced schema, freezes its structural contract, and calibrates mutable field semantics and prompts. Its inference pipeline discovers identities in source order, resolves bounded manifest batches into records, then merges and validates them.
3.1. Problem formulation
- Calibration optimizes a PDF-aware extraction score while constraining the schema structure to remain equal to the induced contract.Gold JSONs establish the contract, and an LLM generates initial field descriptions; prompts and descriptions can change without altering the downstream JSON interface.Source: §3.1 · p. 2
3.2. Prompt roles and mutable schema semantics
- Three prompts manage description patches, source-ordered identity discovery, and full-record resolution, while adaptable descriptions specify field-local evidence, units, missing values, and sample association.Descriptions cannot add, delete, rename, or relocate fields; manifest entries are resolved in non-overlapping slices of up to five.Source: §3.2 · p. 2
3.3. Textual calibration
- TextGrad calibration runs the extractor on training PDFs, uses a PDF-aware evaluator’s scores and root-cause feedback to update prompts, and retains the highest-mean-scoring state.Feedback is aggregated into reusable rules rather than paper-specific facts, and schema patches update descriptions sparsely.Source: §3.3 · p. 2
3. METHOD
- At inference, CPSE discovers and orders material identities, resolves disjoint manifest batches, merges records in source order, and validates them against the fixed contract.The overview depicts schema induction followed by textual calibration and bounded manifest-conditioned resolution before deterministic merging and validation.Source: §3.3 · p. 2, Figure 1 · p. 3
4. EXPERIMENTS
On 17 held-out polymer-science papers, CPSE outperformed an execution-matched baseline, with gains also appearing under an independent judge and in a blinded expert audit. The experiments examine the data and evaluation setup, adaptation variants, and sensitivity to training subsets and document-level cases.
4.1. Data and Calibration Setup
- The study used three randomly selected papers for induction and calibration, leaving 17 of 20 expert-annotated papers for held-out evaluation.The induced structural contract contained 478 typed nodes; individual papers contained 2–54 material records.Source: §4.1 · p. 3
4.2. Evaluation and Comparison Setup
- Two PDF-aware LLM judges scored held-out predictions, and a blinded expert audit compared three methods on eight category-balanced evidence items from each of ten randomly selected papers.Paired comparisons averaged three runs per document; Terra served as a held-out check because Sol provided the calibration objective.Source: §4.2 · p. 3
4.3. Main Results and Ablations
- +10.29 Sol points for joint prompt–description adaptation over the shared single-pass baseline, versus +6.81 for prompt-only and +5.64 for description-only adaptation.The internal ablations report scores on 17 held-out PDFs.Source: §4.3 · pp. 3–4, Table 1 · p. 4
- +9.93 points over the execution-matched baseline, with 14 wins, two ties, and one regression across 17 papers.The paired-bootstrap 95% CI was [5.12, 16.11], and the exact sign-flip p = 1.83 × 10−4.Source: §4.3 · p. 4
- +7.21 against the execution-matched baseline under the held-out Terra judge; the blinded audit found 78 fully correct facts versus 62, with eight of ten documents improving and two tying.The Terra 95% CI was [4.35, 10.09]; the audit reported no regressions (two-sided exact sign test, p = 0.0078).Source: §4.3 · p. 4
- GEPA+manifest performed best among evaluated external baselines but remained 5.43/5.59 points below CPSE under Sol/Terra; it attained 67 fully correct audited facts versus CPSE’s 78.Both judges ranked methods identically; the paired-bootstrap 95% CIs for the gaps were [1.16, 9.85]/[1.04, 10.22].Source: §4.3 · p. 4
4.4. Training-Subset Sensitivity and Mechanism Analysis
- Training-subset gains were non-monotonic: +3.48 for one-paper subsets, +2.23 for two-paper subsets, and +9.93 for all three papers.Case analyses reported gains of +49.2 when recovering record boundaries and +20.0 when correcting sample–condition attribution; identity omissions in the manifest could not be recovered downstream.Source: §4.4 · p. 4
5. CONCLUSION
The conclusion presents CPSE as a few-instance calibration method for scientific PDF extraction that adapts field semantics and prompts while preserving the schema’s structural contract. On expert-annotated polymer-science PDFs, it reports gains under automatic and expert evaluation without changing the JSON interface.
5. CONCLUSION
- CPSE jointly calibrates prompts and field descriptions while keeping the schema’s structural contract fixed and using manifest-conditioned resolution to control record boundaries.The conclusion describes this as few-instance schema calibration for scientific PDF extraction.Source: §5 · p. 4
- On expert-annotated polymer-science PDFs, CPSE improves extraction across automatic and expert evaluation without changing the JSON interface.The conclusion frames a few verified examples as sufficient to improve schema execution without altering the downstream output contract.Source: §5 · p. 4
- The paper positions schema calibration as a distinct post-design stage for reliable LLM extraction.This conclusion follows from adapting operational semantics after schema design while preserving the downstream structure.Source: §5 · p. 4