Source-linked AI summary
From Weak Task Specifications to Scientific Extraction Agents: Optimizing Task Construction
Zixiao Dong, Wei Yang, Zihao Liu, Chenshu Li, Longzhang Liu, Tao Tan, Hong Xie
TL;DR
Scientific extraction agents need task-specific configurations, but weak task specifications may not determine them and manual construction is costly. The framework constructs and optimizes schemas, extraction instructions, and training rubrics; on heterogeneous-catalysis literature, Joint-TG achieves the highest mean across four judge–rubric settings.
Problem
Scientific extraction requires task-specific schemas, instructions, and evaluation criteria, but users may not fully specify them in advance and manual construction requires substantial effort and domain knowledge.
Source: §1 · p. 1Method
Results
83.40 for Joint-TG, which leads all four judge–rubric settings; blinded human comparisons prefer it on all 10 papers for record-level quality and 9 of 10 for scientific-context fidelity.
Source: §4.3 · p. 4Takeaways & Limitations
The results support treating task construction itself as an optimization target for scientific extraction from weak specifications.
Source: §5 · p. 4Document-level cases show remaining challenges in mechanistic grounding and evidence coverage.
Source: §4.5 · p. 4
Abstract
from arXiv · showhide
Most methods that optimize LLM prompts and agent workflows assume that task-specific output schemas, extraction instructions, and evaluation criteria are predefined. For scientific extraction agents, however, a short task goal may not fully determine these components, while specifying them manually is costly. We study the upstream problem of constructing the task-specific configuration from a weak specification containing only a short goal and unannotated reference documents. Rather than treating automatic construction as a fixed preprocessing step, our framework constructs a task-specific schema, extraction instructions, and base training rubrics, then keeps schema construction and extraction instructions editable during optimization. Failure-focused updates concentrate textual-gradient feedback on lower-scoring documents, while training-time evaluation criteria adapt to recurring failures. On a heterogeneous-catalysis literature corpus, automatic construction remains improvable, and optimizing both schema construction and extraction instructions performs best across all four judge-rubric settings, with ablations and blinded human evaluation supporting the proposed formulation.
1. INTRODUCTION
Scientific extraction agents must assemble task-specific records from evidence distributed across papers, but users may not know how to specify the required configuration. The paper treats task construction from a short goal and unannotated documents as an optimization problem, rather than a fixed preprocessing step.
1. INTRODUCTION
- Existing prompt and program optimizers typically improve components within an already instantiated task-specific system; this work instead addresses constructing that system from a weak specification.Source: §1 · p. 1
- Automatic task construction generates a task-specific schema, extraction instructions, and base training rubrics from a weak task specification.Source: §1 · p. 1
- The proposed framework optimizes task construction from weak specifications, keeping schema construction and extraction instructions editable while adapting training rubrics to recurring failures.A weak specification is a short goal paired with unannotated reference documents, without a predefined task configuration.Source: §1 · p. 1
- Across all four judge–rubric settings, sequential optimization of schema construction and extraction instructions performs best, with ablations and blinded human evaluation supporting the result.Source: §1 · p. 1
2. RELATED WORK
Prior work develops scientific extraction systems and optimizes prompts, programs, or agent components. This paper distinguishes its approach by making task construction itself an optimization target under weak specifications.
2. RELATED WORK
- Full-document extraction remains challenging because relevant evidence can be distributed across a paper; Collage and AgentCAT address this through document-aware pipeline design and an evidence-grounded agent workflow, respectively.Source: §2 · p. 1
- Prompt and program optimizers revise instructions or multi-stage programs, while newer methods optimize agent-level components; these approaches assume an instantiated task-specific program.Examples include ProTeGi, OPRO, DSPy, MIPRO, TextGrad, AgentOptimizer, AutoPDL, and GEPA.Source: §2 · pp. 1–2
- DynamicRubric co-evolves an evaluator and policy, whereas this framework adds recurring-failure criteria to stage-specific training rubrics and selects checkpoints using a frozen final-round rubric on held-out documents.Source: §2 · p. 2
3. METHOD
The method constructs a scientific extraction policy from a short task goal and unannotated reference documents, then optimizes its schema-construction and extraction prompts. It focuses updates on lower-scoring documents, adapts training rubrics to recurring failures, and selects checkpoints under a frozen rubric.
3. METHOD
- The initial policy is built from a short goal and unannotated reference documents, with a runtime schema, extraction prompts, and stage-specific base training rubrics.The schema-construction policy produces the schema, while the extraction prompts are generated independently of the initial schema.Source: §3 · p. 2, §3.1 · p. 2, §3.2 · p. 2
3.1. Schema construction from weak specifications
- Schema construction regenerates the schema from editable construction prompts, using the task goal for intent and reference documents for grounded refinement.The initial process maps the goal to a provisional framework, compiles a provisional schema, then refines it against the reference documents.Source: §3.1 · p. 2
3.2. Extraction-prompt generation and execution
- Extraction prompts are generated from the goal independently of the initial schema; execution gathers evidence, maps it to structured values, and checks values against the full PDF.The execution workflow uses candidate generation, evidence-only resolution, and PDF-grounded review before assembling the final record.Source: §3.2 · p. 2
3.3. Failure-focused textual-gradient optimization
- Textual-gradient updates focus on the lower-scoring half of documents, while adaptive criteria can be added for recurring or consequential failures.Each stage starts from a base rubric; the active adaptive criteria have bounded capacity and can be refreshed or retired.Source: §3.3 · p. 2
3.4. Frozen-rubric checkpoint selection
- Checkpoints are rescored on held-out documents under a frozen final-round rubric, with schema and extraction checkpoints selected separately.Using one common rubric avoids comparing scores produced under different evolving training rubrics.Source: §3.4 · p. 2, Figure 1 · p. 3
4. EXPERIMENTS
Experiments on heterogeneous-catalysis papers compare automatically constructed and optimized extraction agents using four judge–rubric settings and blinded human checks. Joint-TG achieves the highest mean across the settings, while ablations and error analysis examine optimization choices and remaining challenges.
4.1. Experimental Setup
- The held-out target corpus contains 50 papers; schema refinement uses 12 PDFs, and each optimization stage uses four update and two selection documents.The corpus spans experimental catalysis, spectroscopy, density-functional theory, mechanistic analysis, and kinetic modeling.Source: §4.1 · p. 3
4.2. Evaluation
- Two expert-designed rubrics score record-level extraction quality and scientific-context fidelity, while two blinded PDF-aware judges evaluate the same extraction files.Ten papers spanning five study types also receive blinded human spot-checks.Source: §4.2 · p. 3
4.3. Main results
- 83.40 for Joint-TG versus 78.37 for Auto-init: Joint-TG leads all four judge–rubric settings on the complete 50-paper target corpus.Its four-score mean improves by 5.03 points over Auto-init; blinded human assessors prefer Joint-TG on all 10 papers for record-level quality and 9 of 10 for scientific-context fidelity.Source: Table 1 · p. 4, §4.3 · p. 4
4.4. Optimization-protocol ablations
- With adaptive training and frozen-rubric selection, using lower-scoring documents yields Judge-T R1 84.50 and R2 84.38, versus 81.91 and 80.29 using all diagnoses.Adaptive criteria also raise R1 from 82.15 to 84.50 and R2 from 81.98 to 84.38; held-out checkpoint selection adds 2.07 and 3.28 points over final checkpoints. Across five rounds, three supplemental criteria remain active, addressing cross-record references, experimental scope and field semantics, and output integrity.Source: §4.4 · p. 4
4.5. Gain and error analysis
- Joint-TG matches or exceeds Auto-init across all eight rubric dimensions, with gains in information recovery and preserving relations among values, conditions, and scientific claims.Document-level diagnostics still find mechanistic-grounding and evidence-coverage errors, including a misattributed literature-derived barrier and an omitted part of an energy map.Source: §4.5 · p. 4
5. CONCLUSION
The conclusion presents task construction from weak specifications as an optimization target for scientific extraction agents. On a 50-paper catalysis corpus, Joint-TG leads the four judge–rubric settings and receives favorable blinded human evaluations.
5. CONCLUSION
- The framework constructs schemas, extraction instructions, and base training rubrics from a short goal and unannotated reference documents, then refines schema construction and instructions while adapting criteria to recurring failures.Source: §5 · p. 4
- On a 50-paper catalysis corpus, Joint-TG achieves the highest mean across all four judge–rubric settings, and blinded human evaluation favors it over Auto-init and AgentCAT.Automatic construction is competitive with manually specified baselines, and both schema construction and extraction instructions remain improvable.Source: §5 · p. 4
- The results support treating task construction itself as an optimization target for scientific extraction from weak specifications.Source: §5 · p. 4