PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes

Simran Arora, Brandon Yang, Sabri Eyuboglu, Avanika Narayan, Andrew Hojel, Immanuel Trummer, Christopher Ré

arXiv:2304.09433v3cs.CL

TL;DR

The paper asks whether large language models can help automatically turn heterogeneous data lakes into queryable structured tables while balancing cost, quality, and generality. It proposes EVAPORATE, comparing direct extraction with code synthesis and improving the latter through candidate-function ensembling with weak supervision; EVAPORATE-CODE+ outperforms direct extraction by 10.1 F1 points.

  • Problem

    Organizations need general systems that automatically convert heterogeneous data lakes into queryable structured tables, while balancing cost, quality, and generality.

  • Method

    EVAPORATE uses LLM in-context learning for direct extraction or code synthesis, while EVAPORATE-CODE+ ensembles many generated extraction functions using weak supervision.

  • Results

    10.1 F1 points (18%) is the advantage of EVAPORATE-CODE+ over EVAPORATE-Direct in the reported evaluation.

  • Takeaways & Limitations

    LLM-based structured-view generation can exploit structural redundancies in semi-structured documents while addressing the cost-quality tradeoff through code synthesis and aggregation.

  • Takeaways & Limitations

    Evaporate-Direct processes tokens linearly in data-lake size, limiting its applicability to large, recurring workloads and dynamically changing data lakes.

Abstract

from arXiv · show

A long standing goal of the data management community is to develop general, automated systems that ingest semi-structured documents and output queryable tables without human effort or domain specific customization. Given the sheer variety of potential documents, state-of-the art systems make simplifying assumptions and use domain specific training. In this work, we ask whether we can maintain generality by using large language models (LLMs). LLMs, which are pretrained on broad data, can perform diverse downstream tasks simply conditioned on natural language task descriptions. We propose and evaluate EVAPORATE, a simple, prototype system powered by LLMs. We identify two fundamentally different strategies for implementing this system: prompt the LLM to directly extract values from documents or prompt the LLM to synthesize code that performs the extraction. Our evaluations show a cost-quality tradeoff between these two approaches. Code synthesis is cheap, but far less accurate than directly processing each document with the LLM. To improve quality while maintaining low cost, we propose an extended code synthesis implementation, EVAPORATE-CODE+, which achieves better quality than direct extraction. Our key insight is to generate many candidate functions and ensemble their extractions using weak supervision. EVAPORATE-CODE+ not only outperforms the state-of-the art systems, but does so using a sublinear pass over the documents with the LLM. This equates to a 110x reduction in the number of tokens the LLM needs to process, averaged across 16 real-world evaluation settings of 10k documents each.

1 INTRODUCTION

Evaporate uses LLMs to generate structured views from heterogeneous documents without customization, exploring a cost-quality tradeoff between direct extraction and synthesized code. Its Code+ variant ensembles many candidate functions with weak supervision, improving quality while reducing LLM processing costs.

  • Motivation: Prior systems rely on format-specific assumptions, linguistic tools, human-written rules, or domain annotations, limiting generality across heterogeneous documents.The target documents include HTML, PDFs, and text, while prior approaches often focus on HTML or require labeled data.
  • Evaporate: Evaporate automatically identifies a schema and extracts values into tables without customization, training, or human effort across diverse settings.The system targets semi-structured data lakes and supports multiple document formats.
  • Cost-quality tradeoff: 24.9%: Evaporate-Code underperforms Evaporate-Direct by 13.8 F1 points on average across the evaluation settings.Code synthesis reduces processing costs, but its synthesized functions vary in quality.
  • Cost-quality tradeoff: Evaporate-Direct prompts an LLM to extract values from each document, while Evaporate-Code synthesizes reusable extraction code.Direct extraction can be competitive with state-of-the-art systems, whereas code synthesis applies generated functions at scale.
  • Code synthesis + aggregation: Evaporate-Code+ synthesizes many candidate functions and ensembles their extractions using weak supervision to reduce quality variance.The method addresses generated functions that work only on narrow document slices or contain syntactic and logical errors.
  • Evaluation: 110x: Evaporate reduces LLM token-processing costs at 10k documents per evaluation setting, while Code+ exceeds Direct by 10.1 F1 points.Across 16 settings, Evaporate outperforms learned baselines by 3.2 F1 points end-to-end and 6.7 F1 points on extraction.

2 PRELIMINARIES

This section defines structured-view generation as converting heterogeneous semi-structured documents into queryable tables, while emphasizing tradeoffs among cost, quality, and generality. It positions LLMs as a way to improve generality without domain-specific training or customization.

  • Structured-view systems convert heterogeneous documents such as HTML, PDFs, and text into tables by identifying attributes and extracting records.
  • Existing systems: Prior systems commonly rely on manual labeling, prompt tuning, format-specific assumptions, linguistic tools, or HTML structure.
  • Quality measurement: Quality is evaluated by comparing generated and manually curated tables using Pair F1, with preference for high-coverage attributes.
  • Generality: Generality is difficult because parsing, named-entity, and part-of-speech methods degrade on semi-structured data and long documents, while HTML-specialized systems do not support other formats.
  • LLM background: LLMs support in-context learning through natural-language prompts containing task descriptions and demonstrations.

3 EVAPORATE: A PROTOTYPE SYSTEM POWERED BY LANGUAGE MODELS

Evaporate uses task-agnostic LLM prompting to generate structured views without domain-specific training or prompt customization. Its direct and code-based implementations expose a cost-quality tradeoff, while Code+ uses weak supervision to improve quality at low cost.

  • Evaporate accepts heterogeneous documents and automatically produces a structured view without domain-specific training or prompt customization.
  • Evaporate-Direct: Evaporate-Direct prompts the LLM to identify the schema and extract values from every document using a general prompt template.
  • Evaporate-Direct: Evaporate-Direct matches or exceeds baseline quality in 8 of 16 settings, but its token processing scales linearly with data-lake size, O(n).
  • Evaporate-Code: Evaporate-Code samples documents, synthesizes data-lake-specific extraction functions, and applies them at scale, making LLM token processing fixed with respect to document count.
  • Evaporate-Code: Evaporate-Code tables are 21.9 Pair F1 points worse than Evaporate-Direct on SWDE datasets, illustrating a cost-quality tradeoff.
  • Evaporate-Code+: Evaporate-Code+ synthesizes many candidate functions and ensembles their extractions using weak supervision to improve quality while keeping costs low.
  • Evaporate-Code+: Evaporate-Code+ processes a document-independent number of tokens up to a constant factor determined by the number of generated candidates.
  • Evaporate-Code+: Evaporate-Code+ achieves the highest quality, outperforming Evaporate-Direct by 12.1 F1 points (22%) on average and Evaporate-Code by 25.1 F1 points.

4 EVALUATIONS

Across 16 settings, Evaporate-Code+ reduces LLM processing cost while improving quality over direct extraction and state-of-the-art baselines, without training or customization.

  • Cost: 110x fewer tokens on average makes Evaporate-Code+ substantially cheaper than Evaporate-Direct across the evaluated settings.The reduction is measured for 10k documents per setting; the true benchmark-size reduction is 378x.
  • Quality: 10.1 F1 points (18%) higher quality than Evaporate-Direct shows that aggregating synthesized functions can outperform per-document extraction.The evaluation attributes the improvement to function synthesis combined with aggregation.
  • Comparisons: 3.2 F1 points (6%) over state-of-the-art when generating tables from scratch and 6.7 points (10%) for predefined attributes demonstrate stronger benchmark performance.Evaporate-Code+ uses six natural-language prompts and no training while supporting broader settings than the baselines.
  • Cost: Function-generation cost depends on attributes rather than documents, with crossover points near 40 documents and 2,500 attributes.Direct extraction instead has document-dependent cost and constant cost with respect to attributes in the described comparison.
  • Aggregation: Weak supervision with filtering improves average quality by 7.1 F1 points over unfiltered weak supervision, while abstention adds 1.9 points.The gains vary by setting, reaching 13.8 points on SWDE University and 7.8 points on FDA, respectively.
  • Model robustness: Evaporate-Code+ and Evaporate-Direct remain competitive across four models, although quality decreases for claude and jumbo relative to text-davinci-003 and gpt-4.Their relative quality relationship remains similar across models.

5 RELATED WORK

Prior systems commonly depend on domain, document-format, syntactic, or training assumptions, whereas Evaporate targets heterogeneous documents through a domain- and format-agnostic LLM interface.

  • Prior systems: Existing structured-view systems often assume a single format, especially HTML, and use DOM positions, linguistic tools, or heuristic rules to extract attributes.These assumptions limit applicability to heterogeneous documents and unstructured text.
  • Prior systems: State-of-the-art web extraction methods use site-specific training or assumptions about attribute locations in the HTML-DOM.The paper contrasts these domain- and format-specific approaches with its domain- and document-format-agnostic goal.
  • LLM-based data management: Evaporate processes heterogeneous data lakes with an alternate LLM tradeoff space instead of processing every document solely to extract values.The related approach most closely considered processes every document with the LLM, whereas Evaporate also synthesizes code.
  • LLM-based data management: Unlike relational-data approaches with manually designed prompts, Evaporate supports varied document formats for structured extraction.The cited prior approaches focus on relational data only.
  • Data programming: Evaporate extends weak supervision by automatically generating open-ended extraction functions and using abstention and filtering to manage function quality.This differs from prior automated weak-supervision work focused on classification tasks and human-designed functions.

6 CONCLUSION

Evaporate uses LLM in-context learning to generate structured views from heterogeneous data lakes while exploring the cost, quality, and generality tradeoffs of direct and code-based processing.

  • Conclusion: Evaporate combines LLM in-context learning with synthesized and aggregated code snippets to generate structured views of semi-structured data lakes.The code-based approach exploits structural redundancies across document corpora and uses weak supervision for aggregation.
  • Conclusion: The system is validated across 16 data settings spanning 5 domains and 3 document formats, evaluating cost, quality, and generality.The conclusion frames these dimensions as the central evaluation scope.
  • Conclusion: The study identifies a tradeoff between directly processing data with an LLM and synthesizing code that performs the processing.It also presents an algorithm and theoretical analysis for aggregating the synthesized code outputs.

A.1 Evaluation Protocol

The evaluation reports OpenIE with Pair F1 and ClosedIE with Text F1, compares generated tables with curated ground truth, and fixes the prediction size to the benchmark’s gold-attribute count.

  • Comparison protocol: Evaporate is evaluated on the same SWDE examples as the baselines without cross-validation because it requires no training data.The baselines train on in-domain documents and therefore use website-wise cross-validation.
  • Metrics: Pair F1 evaluates exact agreement between predicted and gold (subject, predicate, value) tuples for OpenIE.The subject is the document filename, while the predicate and value are the extracted attribute and attribute value.
  • Comparison protocol: Users can stop after any number of attributes, but reported performance uses the number of gold attributes for fair comparison.This differs from prior systems that extract either all or no tuples.
  • Metrics: Text F1 evaluates token-wise overlap between predicted and gold strings for ClosedIE.The implementation follows the standard extractive-task metric described in the evaluation protocol.

B DATASET CONSTRUCTION

The benchmarks cover heterogeneous document sources and are constructed with manually defined attributes, extraction functions, and ground-truth corrections. The weak-supervision setup then combines outputs from multiple extraction functions to infer attribute values without labeled data.

  • Dataset construction: The benchmark suite includes FDA medical-device PDFs, NBA player pages from Wikidata, Enron email metadata, and HTML from SWDE and SWDE Plus.SWDE Plus extends SWDE with more diverse attributes beyond root elements of HTML bodies.
  • Attribute and ground-truth construction: Five database graduate students identified attributes useful for analysis and present in at least a majority of documents, retaining those on which all agreed.For NBA documents, the final set contained 19 attributes whose extraction functions were manually written and corrected by review.
  • Attribute and ground-truth construction: Compound attributes were additionally parsed into atomic attributes and values to provide ground truth for schema cleaning.The example includes multiple values such as birth date and location under a single attribute.
  • Weak supervision: Weak supervision learns a label model from extraction-function outputs and infers a final attribute value without directly observing the true label.The model uses observable labeling and agreement behavior while learning source parameters without access to y.
  • Weak supervision: The source model represents function accuracy, labeling frequency, and pairwise agreement, then learns its parameters by optimizing the observed outputs.The parameters can be learned by minimizing negative log marginal likelihood with SGD or by using a closed-form solution.

C.1 Theoretical Analysis

The analysis studies how weak supervision estimates candidate-function accuracies when direct LLM extractions provide noisy labels. It gives a probabilistic guarantee under assumptions and evaluates the gap between ground-truth and LLM-based scoring.

  • Setup: Evaporate-Code+ uses the LLM to generate candidate extraction functions and directly extract values from a small development set.Each candidate function is applied to the development documents, while direct LLM outputs provide comparison labels.
  • Setup: Candidate-function accuracy is estimated by comparing each function’s outputs with the LLM’s direct extractions on the development set.The resulting estimates are used to score functions for weak-supervision aggregation.
  • Theoretical guarantee: The analysis accounts for noisy LLM-derived labels by treating their error rate e as an additional source of estimation error.As e increases, more development examples are needed for accurate function-accuracy estimates.
  • Theoretical guarantee: Under independent data and randomly occurring errors, weak supervision estimates function accuracies within γ with probability 1 − δ when each function labels enough examples.The proposition assumes bounded measured error and a minimum number of labeled datapoints per function.
  • Implications: Increasing the number of aggregated functions requires a larger development set, whereas a more accurate underlying LLM permits a smaller one.This describes the tradeoff governing development-set size in the weak-supervision analysis.
  • Empirical validation: Using ground-truth labels instead of direct LLM outputs yields an upper-bound quality that is 5.1 F1 points higher.The comparison uses a 10-document development set and measures the gap in Algorithm 1’s scoring process.

D.2 Varying Number of Candidate Functions

Evaporate-Code+ generates diverse candidate extraction functions and aggregates their outputs with weak supervision. Quality improves as both the seed-document pool and the number of aggregated candidates increase.

  • Candidate-function aggregation: Evaporate-Code+ generates a diverse pool of candidate extraction functions with variable quality, then aggregates their outputs using weak supervision.The main experiments score candidates on 10 documents and pass the top 10 functions to the weak-supervision algorithm.
  • Varying seed documents: The study varies the number of seed documents used to generate and score candidate functions across 1, 3, 5, and 10 documents.These settings are evaluated for their effect on system quality.
  • Varying candidate functions: The study varies the number of candidate functions passed to weak supervision across k ∈ {1, 3, 5, 10}.The candidate pool is generated and scored using 10 seed documents in these runs.
  • Findings: Quality increases with both the number of seed documents and the number of candidate functions supplied to weak supervision.This establishes that additional candidate-generation evidence and aggregation capacity improve measured quality.

E.1 Additional Baselines from Prior Work

Additional baseline analyses show that general information-extraction systems struggle to produce consistent structured views from heterogeneous documents. OpenIE6 misses relevant attributes in complex formats and returns many unprioritized relations, while span extraction is evaluated as a separate closed-information-extraction baseline.

  • OpenIE: OpenIE6 is a state-of-the-art OpenIE system for unstructured non-HTML text, evaluated qualitatively here because it was not designed for heterogeneous structured views.The comparison extends beyond the document-level information-extraction baselines in the main evaluation.
  • OpenIE: OpenIE6 handles well-formatted sentences but struggles with heterogeneous types, including HTML and complex PDFs where full sentences are difficult to extract.On a SWDE HTML example, it misses the student body size attribute and value.
  • OpenIE: OpenIE6 produces many relations without prioritizing relevant attributes or enforcing consistency across documents.For one FDA document, it extracts 427 relations, including 184 with confidence above 0.5.
  • ClosedIE: The closed-information-extraction comparison evaluates DeBERTaV3 Large, fine-tuned on SQuAD 2.0, for extracting ground-truth attribute values from relevant paragraphs.The model achieves 90.8 F1 on the SQuAD 2.0 development set.

E.2 Efficiency Comparisons Between Evaporate and Baselines

The efficiency comparison examines inference costs across Evaporate and baselines, including how document count and attributes affect method choice. It also reports extensions for source recovery, schema cleaning, and domain-specific prompting.

  • Model and training costs: Evaporate-Code+ and DOM-LM differ in model and training assumptions: Evaporate uses a 175B model, whereas DOM-LM uses a 125M RoBERTa-Base backbone and domain-specific training.DOM-LM underperforms Evaporate despite using task-specific labeled training data, while Evaporate uses none.
  • Inference costs: Evaporate-Code+ requires O(m) inference calls for n documents, while DOM-LM and Evaporate-Direct require O(n) calls.Generated extraction functions can then be applied across the corpus.
  • Inference costs: Inference FLOP costs for DOM-LM and Evaporate-Code+ are in the same order of magnitude, but the cheaper method depends on document source and attribute count.Users can select the more efficient method for their setting using the cost framework.
  • Extensions: TF-IDF with K-means recovered mixed document sources perfectly, achieving an adjusted rand index (ARI) of 1.0 without labeled data or supervision.The experiment mixed Movie and University documents from distinct websites, including sources from the same domain.
  • Extensions: Cleaning Wiki NBA outputs decomposed 19 complex attributes into 35 atomic attributes, while domain-specific demonstrations improved Movies Schema Identification Recall@K by 4.5 points (8%).The domain-specific prompt was applied uniformly across all 8 SWDE Movies websites.

F.4 Discussion of Future Directions

The discussion identifies future directions for reducing LLM costs, expanding function synthesis, improving extraction quality, and adapting LLMs efficiently. It also notes that generated functions may themselves invoke other models or APIs, complicating cost analysis.

  • New applications: Function synthesis may mitigate LLM costs for high-throughput data-wrangling tasks beyond materializing structured views.The authors suggest exploring whether code synthesis can enable general low-cost solutions.
  • New function types: Future function-synthesis systems could generate functions that invoke NLTK, Huggingface, or OpenAI APIs, making their costs harder to characterize.This challenges the assumption that generated functions are inexpensive.
  • Improving quality: Iterative function generation could use compilation errors or high-quality LLM responses to prompt improved extraction functions.The proposed direction would provide feedback when generated code fails to compile or scores poorly.
  • LLM adaptation: Parameter-efficient fine-tuning and retrieval-augmented generation are complementary approaches for adapting LLMs more efficiently or updating their information.Parameter-efficient fine-tuning updates few parameters, while retrieval-augmented generation incorporates retrieved information into generation.

G.1 Case Study of Function Generation and Aggregation

The case study shows how Evaporate generates multiple extraction functions, aggregates their outputs with weak supervision, and returns consolidated values for FDA and Wikipedia examples. The examples also illustrate that aggregation can handle inconsistent function predictions but remains difficult for some attributes.

  • Candidate function generation: Evaporate generates multiple candidate functions for extracting attributes from FDA reports and Wikipedia documents.The examples include regular-expression and HTML-parser functions for 510(k) numbers and player birth information.
  • Function disagreement: For the FDA 510(k) example, 6 of 10 functions miss K143467.txt, while another predicts the incorrect value “K2” for K162526.txt.These disagreements motivate modeling function predictions rather than relying on simple majority voting.
  • Weak supervision: Weak supervision learns from the matrix of candidate-function outputs across documents and predicts which outputs to return to the user.The FDA example uses 100 documents and 10 candidate functions; the model handles varying document-level prediction sets through fixed classes.
  • Wikipedia outputs: The Wikipedia birth-date example produces varied candidate outputs, and the authors characterize the resulting attribute as difficult with lower-quality results.The candidate predictions include dates, years, partial dates, and surrounding biographical text.
Loading 2304.09433v3…