Source-linked AI summary
ScheMatiQ: From Research Question to Structured Data through Interactive Schema Discovery
Shahar Levy, Eliya Habba, Reshef Mintz, Barak Raveh, Renana Keydar, Gabriel Stanovsky
TL;DR
Research questions over large document collections often require structured evidence, but manual schema design and corpus labeling are slow, expensive, and error-prone. ScheMatiQ uses backbone-LLM calls and an interactive interface to discover schemas and extract grounded datasets, recovering most human schemas while adding useful fields in law and computational biology.
Problem
Answering natural-language research questions over document collections requires structured data, traditionally obtained through expensive, slow, and error-prone schema design and corpus annotation.
Method
ScheMatiQ uses backbone-LLM calls to identify observation units, induce question-specific schemas, extract grounded structured data, and support iterative expert revision through a web interface.
Results
ScheMatiQ recovers all but two broad miscellaneous fields in both evaluated domains, while its proposed fields receive mean relevance ratings of 4.2/5 in computational biology and 3.6/5 in law.
Takeaways & Limitations
ScheMatiQ produces meaningful schemas and supports practical research workflows in empirical legal research and computational biology.
Takeaways & Limitations
The evaluation relies on closed-source LLM APIs, making full reproducibility difficult and allowing small run-to-run changes in column names or extracted values.
Abstract
from arXiv · showhide
Many disciplines pose natural-language research questions over large document collections whose answers typically require structured evidence, traditionally obtained by manually designing an annotation schema and exhaustively labeling the corpus, a slow and error-prone process. We introduce ScheMatiQ, which leverages calls to a backbone LLM to take a question and a corpus to produce a schema and a grounded database, with a web interface that lets steer and revise the extraction. In collaboration with domain experts, we show that ScheMatiQ yields outputs that support real-world analysis in law and computational biology. We release ScheMatiQ as open source with a public web interface, and invite experts across disciplines to use it with their own data. All resources, including the website, source code, and demonstration video, are available at: www.ScheMatiQ-ai.com
1 Introduction
ScheMatiQ addresses the costly, error-prone task of converting research questions over document collections into structured, verifiable data. It combines LLM-based discovery and extraction with interactive expert refinement, and evaluations in law and computational biology show recovery of human schemas plus useful new attributes.
- Research questions over document collections require structured data organized around observation units implied by the question and corpus.
- Manual workflows require experts to design schemas and annotators to label the corpus, making structured-data construction expensive, slow, and vulnerable to human error.
- ScheMatiQ uses backbone-LLM calls to identify observation units, induce schemas, and generate grounded structured databases that experts can inspect and revise.
- In law and computational biology, ScheMatiQ recovers the vast majority of human-annotated schemas and introduces new attributes that experts find useful.
- The system is released open source through a public web interface for researchers to use with their own questions and document collections.
2 ScheMatiQ Principles
ScheMatiQ is designed around query-driven discovery, human control, and grounded outputs. These principles connect schema construction to the expert’s question while keeping revisions and evidence inspection available throughout the workflow.
- Query-Driven Discovery: Query-driven discovery grounds the pipeline in the expert’s natural-language question, allowing the same documents to yield different observation units and data structures.
- Human-in-the-Loop: Human-in-the-loop design keeps experts in control by making every component editable and incorporating their domain knowledge at each stage.
- Grounded and Traceable Outputs: Grounded and traceable outputs link each system output to source documents so experts can verify results, assess extraction quality, and trace unexpected outputs.
3 ScheMatiQ
ScheMatiQ converts a research question and document collection into a structured dataset by discovering the observation unit, iteratively building a schema, and extracting evidence-supported values. Experts can revise these stages and the resulting table as their analysis develops.
- Observation Unit Discovery: The pipeline first identifies the observation unit, defining what object each data instance represents and therefore the structure of the resulting dataset.
- Observation Unit Discovery: Different questions over the same documents can imply different observation units, data structures, and document–observation-unit relationships.
- Observation Unit Discovery: An LLM identifies the observation-unit type from the question and document batch, while experts can revise the prediction or specify the type manually.
- Schema Discovery: Schema discovery builds attributes relevant to the research question by iteratively processing document batches and asking whether fields should be added or refined.
- Schema Discovery: Experts can edit, add, remove, or merge fields and incrementally introduce documents while preserving the existing schema.
- Structured Data Extraction: Extraction creates a table whose rows represent observation-unit instances and columns represent schema attributes, with values restricted to clearly supported document evidence.
- Structured Data Extraction: Experts can correct or refine extracted cells and add documents so the table expands with accurate, evidence-supported values.
4 System Evaluation
ScheMatiQ is evaluated against human-annotated schemas and datasets in legal and computational-biology use cases. It largely recovers existing schema fields, adds relevant fields, and extracts observation units and values accurately, with errors concentrated in dense documents, normalization, and missing explicit evidence.
- Experimental setup: The evaluation compares ScheMatiQ with human annotations in legal and computational-biology use cases, measuring schema agreement, omissions, and novel fields.The benchmarks derive from prior large-scale annotation projects with existing corpora, research questions, schemas, and annotated datasets.
- Schema recovery: ScheMatiQ recovers all but two broad miscellaneous fields and adds relevant fields, rated 4.2/5 in computational biology and 3.6/5 in law.Examples of newly suggested legal fields include legal basis, injunction scope, and the presidential administration whose policy was challenged.
- Schema inputs: Combining the research question with documents yields richer, context-specific schemas than either input alone.Question-only schemas tend to be generic, while document-only schemas can be broad and misaligned; no three-way overlap was observed.
- Observation-unit recovery: ScheMatiQ identifies 87% of proteins and 97.5% of judges, with 82% precision on the extracted judge set.Most misses occur in documents containing many observation units, while recall is near-perfect when documents mention a single entity.
- Value extraction: Value extraction is generally precise but loses recall when values require normalization or lack explicit textual evidence.Biology errors often reflect surface forms instead of canonical names, and omitted values commonly explain lower recall rather than incorrect extraction.
- Value extraction: In law, court level is extracted with no errors and decision date with 97% accuracy, while decision/vote remains the main error area.The legal analysis covers 143 successfully aligned judges; some decision-date disagreements are attributable to errors in the human-annotated dataset.
5 Related Work
Prior schema-discovery methods either learn document-to-table mappings without queries or rely on predefined schemas and generic comparison intents. ScheMatiQ instead conditions discovery on both the research question and documents, producing question-specific structures that experts can revise.
- Earlier text-to-table methods learn document-to-table mappings from supervised pairs without conditioning on a query.
- Literature-review systems generate tables whose rows are papers and columns capture comparison aspects, optionally using a user-provided intent.
- Some approaches mine reusable ontologies from domain specifications, while others assume a predefined schema for scientific extraction.
- ScheMatiQ conditions schema discovery on both the research question and documents and explicitly identifies the observation unit defining each row.
- Experts can revise the observation unit, schema, and extracted values at any stage, supporting specific research questions rather than generic document comparison.
6 Conclusion
ScheMatiQ is an interactive framework for query-driven schema discovery and dataset construction. Evaluations in empirical law and computational biology produced meaningful schemas and supported practical research workflows.
- ScheMatiQ identifies an observation unit, induces a question-specific schema, and extracts a structured dataset that experts can iteratively refine.
- Evaluations across empirical legal research and computational biology show that ScheMatiQ produces meaningful schemas and supports practical research workflows.
7 Limitations and Ethical Concerns
The experiments rely on closed-source LLM APIs, limiting guaranteed reproducibility because outputs can vary across runs and provider updates. Users can otherwise avoid stored session data unless they opt in to research recording.
- Closed-source LLM APIs make full reproducibility difficult to guarantee because runs may vary through nondeterministic decoding or unannounced model updates.These variations may slightly change column names or extracted values; locally hosted open-weight models can mitigate the issue.
- Users can opt in to have their data recorded for research purposes; otherwise, ScheMatiQ does not store session data.
A Use Cases: Full Specifications
The use cases apply ScheMatiQ to legal court decisions and computational-biology papers, with observation units defined as individual judges and proteins. Their schemas capture domain-specific attributes for analysis.
- Legal Domain: The legal dataset contains U.S. court decisions concerning immigration policies and injunction proceedings.
- Legal Domain: The legal question compares judges appointed by Trump, other Republican, and Democratic presidents on immigration injunction decisions.
- Legal Domain: Each individual judge is a legal observation unit, including separate rows for judges serving on the same panel.
- Legal Domain: The legal schema includes panel composition, appointing presidents and parties, policy instruments, legal bases, dates, immigration policies, and decision outcomes.
- Computational Biology Domain: The computational-biology dataset comprises 110 scientific papers describing experimental studies of nuclear export signals in proteins.
- Computational Biology Domain: The computational-biology question asks whether a protein sequence contains a nuclear export signal, its strength, and confidence in that assessment.
- Computational Biology Domain: Each protein or polypeptide sequence evaluated for nuclear export signals serves as the computational-biology observation unit.
- Computational Biology Domain: The computational-biology schema covers motif counts, export mechanisms, critical residues, presence status, evidence, affinity, regulation, localization, conservation, and identified sequences.
B System Architecture
ScheMatiQ separates its system into a frontend, backend, and core library, combining interactive document analysis with reusable extraction components. The architecture supports live updates, multiple model providers, and portable deployment.
- Frontend: The frontend provides interactive query configuration, document upload, schema editing, and extracted-table exploration.It is implemented as a React application using TypeScript and Tailwind CSS, with real-time backend updates.
- Backend: The backend exposes REST endpoints for pipeline operations and streams step-by-step extraction progress through WebSockets.
- Core Library: The core Python library implements observation-unit discovery, schema discovery, and value extraction.It supports GPT-4, Gemini, Together AI models, and locally hosted open-weight models through HuggingFace Transformers.
- System Integration: Separating the core library from the web interface enables programmatic use alongside session management, cloud storage, and human-in-the-loop editing.The full system is deployed with Docker containers on Railway for portability and scalability.
C Prompt Templates
ScheMatiQ’s prompt structures guide its discovery pipeline, which is grounded in the expert’s natural-language query. Figure 7 focuses on observation unit discovery and schema discovery as two core stages.
- Prompt Templates: The prompt structures guide the core discovery pipeline.
- Prompt Templates: Figure 7 presents simplified LLM prompt excerpts for observation unit discovery and schema discovery.