PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Information Extraction with Learned Queries

Omar Sharif, Soroush Vosoughi, Nikhil Singh

arXiv:2608.31058v1cs.CL

TL;DR

Information extraction often relies on questions that fail to expose evidence distributed across documents, so the paper studies question design as a separate bottleneck. It introduces LOQ and FEEDQ to optimize document-specific question sets and transfer them to lightweight generators, achieving large cross-benchmark gains from question improvement.

  • Problem

    Question-based extraction commonly uses generic role-level or template-derived questions, although relevant evidence may be distributed, indirect, or inconsistently expressed.

  • Method

    LOQ uses FEEDQ to iteratively refine document- and role-conditioned question sets, then fine-tunes lightweight question generators on 12,820 optimized examples.

  • Results

    18.6 F1 points: optimized questions outperformed the strongest baseline across four clinical datasets and all prediction models.

  • Takeaways & Limitations

    Question design is a separable component of extraction, and optimized questioning can transfer to lightweight generators for test-time use without ground-truth arguments.

  • Takeaways & Limitations

    FEEDQ uses GPT-OSS-120B as the sole model for generation, refinement, leakage checking, and prediction during optimization.

Abstract

from arXiv · show

When information extraction fails, a natural instinct is to improve the model doing it: for example, by scaling it up or refining its reasoning. In this paper, we show that another part of the pipeline matters at least as much: the queries used to elicit this information. Across four clinical benchmarks and five LLMs, improving the question design alone raises performance by 18.6 F1-score points, i.e. more than using larger extraction models. To make such question design learnable, we introduce List of Questions (LoQ), which generates document-specific question sets, and FeedQ, a feedback-driven optimization method that iteratively refines questions against extraction outcomes. The resulting optimized questions can be used to train lightweight generators: with fine-tuning, 4B-parameter models match or outperform expert-derived baselines and substantially exceed the performance of much larger untuned models. We release a dataset of 12,820 optimized questions to support a broader shift in information extraction research toward treating question design as a first-class problem.

1 Introduction

Information extraction can fail because generic questions miss evidence distributed across clinical narratives, making question design a major bottleneck alongside model capability. LOQ addresses this by learning document-specific question sets and reports substantial gains from question improvement alone.

  • Illustration: In a case report, LOQ’s targeted questions recovered all four adverse drug reactions, whereas a standard single role-level question retrieved two.
  • Motivation: Clinical information is often distributed across sentences, expressed indirectly, or intertwined with narrative context.
  • Motivation: Question-based extraction asks an LLM to generate arguments for a target role, but generic role-level questions may systematically miss arguments.
  • LOQ: LOQ learns extraction questions tailored to each document–role pair and refines candidate questions using extraction quality feedback.
  • Empirical evidence: 18.6 F1 points: improving questions increased average performance across four clinical benchmarks and five LLMs.
  • LOQ: FEEDQ-generated questions form a dataset of 12,820 document, role, and question-set triples for training a lightweight question generator.

2 Related Work

Prior information extraction methods commonly reuse template-based questions across documents, while recent work improves formulation through contextualization, reinforcement learning, or question combinations. LOQ instead optimizes document- and role-conditioned question sets against extraction outcomes and transfers the resulting behavior to a lightweight generator.

  • Information extraction as question answering: Question-answering formulations support free-text argument extraction, but questions are often written once from annotation guidelines or role templates and reused across documents.
  • Quality of questions: Recent methods improve question formulation with event-specific templates, reinforcement learning, or combinations of contextualized and uncontextualized questions.
  • Quality of questions: Most existing approaches retain a question-per-role paradigm despite roles involving multiple arguments expressed indirectly or across sections.
  • Learning to ask the right questions: LOQ optimizes document- and role-conditioned question sets under a multi-tier extraction objective, then fine-tunes a lightweight generator to produce them without ground-truth arguments.

3 Methods

LOQ separates question optimization, question generation, and prediction. FEEDQ iteratively refines document- and role-specific questions using extraction feedback, then supplies training data for lightweight question generators.

  • LOQ framework: LOQ automates extraction through question optimization, question generation, and prediction phases.FEEDQ first derives gold question sets, a fine-tuned generator produces questions from documents and roles, and a prediction model extracts arguments.
  • FEEDQ: FEEDQ optimizes question sets by evaluating their extracted arguments against ground truth and retaining the highest-F1 set.The loop tracks matched, missing, and over-generated arguments, then refines questions to target misses and reduce over-generation.
  • FEEDQ: FEEDQ begins with zero-shot questions conditioned on the document, role, and arguments, while applying a leakage check before extraction.The leakage module rewrites questions that potentially expose ground-truth arguments.
  • FEEDQ: FEEDQ stops at a target score, after a patience limit without improvement, or at the maximum iteration count, returning the best rather than necessarily final set.F1 is not guaranteed to improve monotonically across iterations.

4 Datasets and Evaluation

The evaluation covers four heterogeneous clinical and biomedical datasets, with held-out test evaluation using F1. Development data supports FEEDQ design, while training data supplies fine-tuning examples.

  • Datasets: The benchmarks span full-length case reports, short pharmacovigilance texts, informal health forums, and PubMed case-report snippets.All datasets are converted into document–role–argument-list triples.
  • Evaluation: Table 1 isolates question quality by holding the prediction model fixed while varying only how questions are formed.It compares zero-shot FEEDQ questions with iteratively refined FEEDQ questions across four datasets and five prediction models.
  • Evaluation: Each dataset is split into training, development, and test sets with distinct roles in optimization, fine-tuning, and final evaluation.The test set remains held out for evaluation on unseen data.

5 Importance of Better Questions

Across datasets and prediction models, extraction performance depends strongly on question quality rather than model capability alone. The evidence also shows that effective questions must be document-specific and, for some roles, multiple and iteratively refined.

  • Impact of question improvement: Contextual-Q raises recall but lowers precision because its unoptimized questions generate many extra candidate arguments.Across datasets, it averages 5.27 questions per document–role pair; DiscourseEE recall rises from 65.7 to 76.3 while precision falls from 37.4 to 28.8.
  • Discovering the right questions: GPT-5.2 scores 76.1 F1 overall for zero-shot question generation, while GPT-OSS-120B reaches 74.9 at over 40× lower token cost.The paper therefore uses GPT-OSS-120B as the question-generation model.
  • Impact of question improvement: 18.6 F1 points separate Optimized-FeedQ from Knowledge-Q across four datasets and all prediction models.Gains range from 11.0 on PHEE to 32.7 on MACCROBAT.
  • Impact of question improvement: 13.2 F1 points are gained by zero-shot FeedQ over Knowledge-Q, while FEEDQ refinement adds another 5.4 F1.The largest refinement gains occur on DiscourseEE (+4.9) and MACCROBAT (+10.6).
  • Optimized-question analysis: A single FEEDQ pass suffices for 87% of PHEE roles and 75% of MACCROBAT roles, but 47% of DiscourseEE and a quarter of CaseReportBench roles need at least two iterations.Five iterations suffice to find the best-scoring question set across all four datasets.
  • Optimized-question analysis: CaseReportBench assigns one optimized question to 41% of roles, while 59% receive at least two and 27% require four or more.DiscourseEE shows a similar need for multiple targeted questions, whereas single questions usually suffice for PHEE and MACCROBAT.

6 Learning the Right Questions

Fine-tuning lightweight Qwen models on FEEDQ-optimized questions substantially improves document-specific question generation, often surpassing human-written and larger untuned baselines. Benefits extend beyond the clinical training benchmarks, though performance varies with dataset structure.

  • 6.1 Tuned models generate effective questions: 12,820 training triples from FEEDQ raise both fine-tuned Qwen3 models to 62.7 mean F1, versus approximately 53 without fine-tuning.The triples contain documents, roles, and optimized question sets.
  • 6.1 Tuned models generate effective questions: The fine-tuned models outperform non-fine-tuned GPT-OSS-120B, GPT-5.2, and Gemini-3.1-Pro across the evaluated question-generation comparison.The largest gains occur on PHEE and MACCROBAT, while the overall improvement is not dataset-specific.
  • 6.1 Tuned models generate effective questions: 62.7 mean F1 exceeds Knowledge-Q’s 61.3 average, although Knowledge-Q remains marginally better on CaseReportBench.Fine-tuned models lead on PHEE and MACCROBAT but trail on some lower-supervision datasets.
  • 6.1 Tuned models generate effective questions: Limited training data constrains learning of dataset-specific question patterns, with CaseReportBench providing 620 samples and DiscourseEE 2,200.PHEE and MACCROBAT each contribute 5,000 training samples.
  • 6.2 Out-of-domain evaluation: Out-of-domain fine-tuning benefits DocEE and MUC4 but not GENEVA, where broader base-model question sets better serve multiple coreferent arguments.Precision-recall analysis indicates that fine-tuned questions improve precision across all three datasets.

7 Conclusion

The paper argues that information extraction depends partly on how queries represent both the problem and the available data. It presents evidence and an automated solution centered on better question design.

  • 7 Conclusion: Poor query selection can put an extractor on the wrong trail and make necessary information invisible.The conclusion frames queries as part of the representation used for complex extraction.
  • 7 Conclusion: The paper offers evidence of the problem and a scalable, automated solution through its methods and released data.It invites information-extraction research to use these resources for improved information access.

Limitations

The paper’s evidence is concentrated in clinical and biomedical extraction, and FEEDQ optimization relies on a single large language model. Out-of-domain experiments suggest broader effectiveness, but wider applicability remains to be demonstrated.

  • Optimizer scope: FEEDQ uses GPT-OSS-120B alone for question generation, refinement, leakage checking, and prediction during optimization.The authors identify varying the optimizer or ensembling multiple models as directions for further study.
  • Domain scope: All four primary benchmarks are clinical or biomedical, with arguments distributed across narrative sections and expressed in domain-specific terms.Three non-clinical out-of-domain datasets were tested, but application to news, law, or finance remains future work.

A Datasets

The datasets are unified as document–role–argument extraction tasks spanning clinical narratives, pharmacovigilance, health forums, and clinical event reports. Their structures vary substantially in density, argument implicitness, role count, and sample availability.

  • Dataset construction: Each unified sample consists of a document, a role, and ground-truth arguments, with 500-example development and test splits except CaseReportBench’s 350.The datasets are converted into a common format for evaluation.
  • Clinical datasets: CaseReportBench contains expert-annotated categories from long, unstructured rare-disease case reports requiring reasoning across entire documents.Categories include laboratory findings, diagnosis, treatment, and outcomes.
  • Clinical datasets: PHEE covers 14 pharmacovigilance roles, including drug, dosage, route, frequency, duration, disorder, and patient demographics.Its original annotations include sentence-level trigger and argument spans.
  • Clinical datasets: DiscourseEE contains 34 roles from opioid-use-disorder health forums, with 51.2% of arguments implicit and inferable from context.This makes it a particularly challenging argument-extraction dataset.
  • Clinical datasets: MACCROBAT contains 22 argument roles from PubMed case reports and averages 3.21 events per sentence.The study uses 6,000 randomly selected samples from 6,882 document-role samples.
  • Models: Evaluation spans Qwen, GPT, and Gemini model families, including open-source and proprietary models at different scales.Qwen3-4B and Qwen3-8B are also fine-tuned with LoRA for question generation.

B.2 FEEDQ Parameters and Leakage Analysis

FEEDQ refines extraction questions iteratively and filters answer leakage before questions enter training. The resulting pipeline uses manually sourced expert questions for comparison and standardized argument matching for evaluation.

  • FEEDQ parameters: 5 iterations suffice to find the best-scoring question set across all four datasets, although dataset complexity changes the required budget.FEEDQ uses patience of 3 and targets a score of 1.0, corresponding to full extraction of ground-truth arguments.
  • Leakage analysis: 42–55% of iterations and 45–65% of samples contain leakage before filtering.The leakage checker flags generated questions containing ground-truth content before those questions are removed.
  • Leakage analysis: The checker’s false negative rate is 0.83% (5/600) after manual inspection of 600 filtered question sets.The inspection covers 150 question sets per dataset.
  • Baselines: Human-written questions are taken from dataset-author materials or minimally adapted from expert annotation guidelines.For DiscourseEE, the released expert questions are used verbatim; other datasets use role definitions from papers or guidelines.
  • Evaluation: Evaluation matches predicted and ground-truth arguments at exact, relaxed similarity, or LLM-as-judge complex levels before computing precision, recall, and F1.Relaxed matching uses SBERT similarity above 0.85, while complex matching uses GPT-OSS-120B.

B.6 Impact of Data Scaling

Question-generation fine-tuning benefits from more training data, but the effect depends on dataset size and complexity. Fine-tuned questions can also generalize beyond the training domains, with targeted questions improving performance under domain shift.

  • Data scaling: Performance rises monotonically with added data on PHEE and MACCROBAT, where even 10% of the training triples beats the base model.Each dataset contains 5,000 triples.
  • Data scaling: 10% fine-tuning data lowers performance on CaseReportBench and DiscourseEE, but performance recovers and surpasses the base model at 100%.These datasets contain 620 and 2,200 triples, respectively; the early drop is consistent with catastrophic forgetting from small in-domain samples.
  • Out-of-domain generalization: Fine-tuning improves Qwen3-8B by up to +7.4 F1 on DocEE under out-of-domain evaluation.The fine-tuned model produces a small set of targeted questions instead of the base model’s large pool of loosely relevant questions.
  • Out-of-domain generalization: GENEVA’s annotation style rewards broader probing because multiple coreferent or syntactically distinct arguments may share one role.Focused fine-tuned questions improve precision but reduce overall F1 through lower coverage on this dataset.
  • Question construction: The question-generation prompts seek concise, nonredundant questions that target one or more arguments while avoiding vague or yes/no phrasing.The refinement prompt uses prior questions and extraction precision, recall, and F1 as feedback.

D Auto-Interpretability

Auto-interpretability groups changes between question sets to characterize how optimized questions differ from vanilla and zero-shot questions. The reported patterns include specificity, scope, terminology, context, and format changes.

  • Analysis procedure: The analysis compares Vanilla→Zero-Shot and Zero-Shot→Optimized question changes using LLM-as-judge labels and clustered embedding differences.Cluster summaries are generated with the matryoshka methodology.
  • Observed transformations: Optimized questions add clinical scope, anatomical and biochemical detail, and more formal terminology.Examples include symptoms, diagnostics, family history, outcomes, interventions, brain regions, metabolites, and laboratory findings.
  • Observed transformations: Optimization refines wording by substituting precise terms, inserting clarifying qualifiers, deleting redundancy, and adjusting grammatical structure.These changes are reported as lexical and structural transformations between question versions.
  • Observed transformations: Question scope shifts toward specific clinical features, entities, terminology, temporal or treatment context, and constrained response formats.The reported transformations include narrowing from broad categories and changing between open-ended and yes/no forms.
  • Observed transformations: Medication-focused transformations generalize drug names and dosages in some cases while modifying dosage, frequency, temporal, and supervision constraints.The analysis also reports shifts in comparison targets and medication-category scope.
Loading 2608.31058v1…