Source-linked AI summary
Can Foundation Models Wrangle Your Data?
Avanika Narayan, Ines Chami, Laurel Orr, Simran Arora, Christopher Ré
TL;DR
The paper asks whether Foundation Models can perform classical data cleaning and integration tasks despite being trained primarily for language generation. It casts five tasks as prompting problems and evaluates large FMs, finding strong few-shot transfer while highlighting prompt, privacy, and integration challenges.
Problem
It is unclear whether Foundation Models can reason over structured data and work out of the box on complex cleaning and integration tasks.
Method
The paper serializes structured data as text and casts five cleaning and integration tasks as natural-language generation tasks with zero- or few-shot prompts.
Results
Large FMs achieve SoTA performance on many data tasks with zero-to-few natural-language demonstrations, including GPT-3-175B outperforming SoTA approaches across five enterprise tasks.
Takeaways & Limitations
The results suggest that language-guided models could support human-in-the-loop data integration across a broader range of data-management tasks.
Takeaways & Limitations
Organizations may be unable to send sensitive data to third-party FM APIs, motivating competitive open-source models.
Abstract
from arXiv · showhide
Foundation Models (FMs) are models trained on large corpora of data that, at very large scale, can generalize to new tasks without any task-specific finetuning. As these models continue to grow in size, innovations continue to push the boundaries of what these models can do on language and image tasks. This paper aims to understand an underexplored area of FMs: classical data tasks like cleaning and integration. As a proof-of-concept, we cast five data cleaning and integration tasks as prompting tasks and evaluate the performance of FMs on these tasks. We find that large FMs generalize and achieve SoTA performance on data cleaning and integration tasks, even though they are not trained for these data tasks. We identify specific research challenges and opportunities that these models present, including challenges with private and domain specific data, and opportunities to make data management systems more accessible to non-experts. We make our code and experiments publicly available at: https://github.com/HazyResearch/fm_data_tasks.
1 INTRODUCTION
The paper investigates whether large Foundation Models can handle classical structured-data cleaning and integration tasks through prompting. It reports strong transfer across five enterprise tasks while identifying prompt sensitivity and broader deployment challenges.
- Large Foundation Models generalize to new tasks without task-specific finetuning, extending their established language-task capabilities toward structured data.
- Existing data-task systems require task-specific architectures, hard-coded knowledge, and substantial labeled data, creating siloed and difficult-to-maintain solutions.
- FMs offer a task-agnostic natural-language interface, encoded knowledge about common entities, and zero- or few-shot use with little labeled data.
- GPT-3-175B outperforms SoTA ML- and DL-based approaches on five enterprise data tasks with few examples, despite only being pretrained for text generation.
- FMs are brittle to prompt-format differences, while manually selected demonstrations improve performance over randomly selected examples.
- Applying FMs in data-management pipelines raises challenges around updating knowledge and handling private, temporal, and local data.
2 BACKGROUND
The background introduces the classical data tasks studied and contrasts pretrained language models with large autoregressive Foundation Models. It emphasizes the latter’s few-shot and zero-shot in-context learning capabilities.
- The paper focuses on entity matching, error detection, and data imputation as structured-data tasks.
- Entity Matching: Entity matching predicts whether entries from two datasets represent the same real-world entity, typically after blocking removes obvious non-matches.
- Error Detection: Error detection identifies attributes whose values are erroneous, commonly using rules, functional dependencies, knowledge bases, or statistical methods.
- Data Imputation: Data imputation infers missing attribute values, while prior approaches struggle to impute values absent from their training data.
- Foundation Models: Large autoregressive language models such as GPT-3 are pretrained to predict the next word and contain billions of parameters.
- Foundation Models: GPT-3-175B can perform some tasks from only a task description or a few examples, without updating model parameters.
3 FOUNDATION MODELS FOR DATA TASKS
The paper adapts structured data tasks to Foundation Models by serializing tables into text and expressing task outputs as generated text. Few-shot demonstrations provide task guidance, but prompt construction and example selection materially affect performance.
- Applying FMs to data tasks requires adapting structured inputs to text, casting tasks as generation, and constructing demonstrations for few-shot prompting.
- Input Serialization: Relevant table attributes are serialized into text, with NULL values represented as empty strings; model performance can depend on serialization and prompt format.
- Task Prompts: Prompts pass serialized records to the FM, which generates Yes/No answers for matching and error detection or a missing value for imputation.
- Task Prompts: The prompting framework uses natural-language templates and can extend beyond the data tasks considered in the paper.
- Task Demonstrations: Task demonstrations teach the model the required output format and finer-grained dataset semantics.
- Task Demonstrations: Random demonstration sampling can produce high performance variance, whereas manually constructed examples improve performance but require more time.
4 EXPERIMENTS
The experiments test whether large foundation models transfer to data cleaning and integration through zero- and few-shot prompting. GPT-3-175B achieves strong benchmark performance, but results depend substantially on task demonstrations, prompt design, and domain familiarity.
- Experimental Setup: The study compares foundation models with state-of-the-art methods across entity matching, error detection, schema matching, data transformation, and data imputation.It evaluates zero- and few-shot settings using established datasets, task-specific metrics, and competitive baselines.
- Few-shot Performance: GPT-3-175B achieves SoTA performance in few-shot settings on 4 entity matching, 2 imputation, 2 data transformation, 2 error detection, and 1 schema matching benchmark datasets.The manually curated demonstrations support performance exceeding fully finetuned PLM approaches for several tasks.
- Zero-shot Performance: In zero-shot imputation, the model outperforms statistical approaches and standard data repair engines, while entity matching performs substantially worse without demonstrations.The entity-matching gap indicates that demonstrations are particularly important for that task.
- Discussion: Large FMs transfer to data tasks despite training on English language rather than tabular-task semantics, and imputation results suggest encoded knowledge helps correct records.The paper gives functional dependencies such as address–ZIP-code relationships as an example of relevant encoded knowledge.
- Limitations: The model struggles with entity matching in domains containing uncommon jargon, including product-specific identifiers in the Amazon-Google dataset.The paper identifies domain-specific data as a boundary for semantic understanding and reasoning.
- Prompt Tuning: Entity matching improves by 13.7 F1 points on average when informative attributes are selected and noisy attributes removed.The ablation evaluates three datasets and finds that attribute selection materially affects accuracy.
- Prompt Tuning: Changing a prompt phrase produces a 9.4 F1-point average gap, while manually curated demonstrations outperform random examples by 14.7 F1 points.Together, the ablations show that formatting and example selection are major sources of performance variation.
- Discussion: Effective prompt tuning requires selecting informative attributes, crafting understandable formats, and constructing instructive demonstrations, making iterative prompt programming a human-in-the-loop process.The paper frames this as a shift from tuning model parameters toward engineering prompts and finding suitable examples.
5 RESEARCH AGENDA
The research agenda outlines opportunities for FMs to unify and simplify data-management workflows, alongside challenges involving integration, debugging, domain specificity, privacy, and prompt automation.
- Opportunities for FMs: FMs could unify siloed data-integration pipelines through a natural-language interface and become central to future data-orchestration workbenches.Their interface can reduce reliance on separate hand-engineered pipelines.
- Opportunities for FMs: FMs may make data-management systems more accessible by shifting effort from labeling and finetuning toward representative natural-language prompts.The authors envision workflows usable by non-machine-learning experts.
- Opportunities for FMs: FMs are most useful for rapid prototyping when training data is scarce, but fully supervised models become preferable once sufficient data is collected.FMs can also label or generate data with human-in-the-loop feedback during prototyping.
- Opportunities for FMs: FMs can learn from unlabeled enterprise data exhaust, including system logs, structured data, and analyst interactions.This passive learning opportunity could improve performance on data tasks.
- Practical Considerations: Integrating FMs into data-management workflows requires translating natural-language specifications into GUI actions and systematically incorporating existing-system outputs.The paper also identifies open questions for replacing systems whose rules are not encoded in FM knowledge.
- Technical Challenges: Non-deterministic errors motivate transparency and debugging mechanisms, while specialized domains may require costly continued pretraining or parameter-efficient adaptation.The paper specifically highlights confidence monitoring, primitive-operation decomposition, and domain adaptation methods.
- Technical Challenges: Privacy constraints can prevent organizations from sending sensitive data to third-party APIs, motivating competitive open-source FMs.The paper identifies prompt ensembling and reframing as possible directions for improving open-source models on data tasks.
- Technical Challenges: Prompt performance is sensitive to schemas and minor formatting changes, making automated prompt construction and example retrieval an open research problem.The paper points to soft prompt tuning and learned in-context-example retrieval as relevant approaches.
6 CONCLUSION
The paper evaluates foundation models on classical data-integration and cleaning tasks and finds that large models can achieve state-of-the-art performance with few natural-language demonstrations. These results suggest potential for language-guided, human-in-the-loop data-management practices.
- Conclusion: Large FMs achieve state-of-the-art performance on many classical data tasks with zero-to-few natural-language task demonstrations.The paper emphasizes transfer without task-specific finetuning despite next-word pretraining.
- Conclusion: The findings point toward language-guided models supporting human-in-the-loop data-integration practices across broader data-management tasks.This is presented as a direction suggested by the reported results.
A SMALL FM FINETUNING EXPERIMENTS
The finetuning experiments compare full and lightweight adaptation of small FMs, highlighting tradeoffs between sample efficiency and training efficiency. Lightweight adapters update only a small portion of parameters, while larger models can avoid finetuning through zero- or few-shot use.
- Finetuning Tradeoffs: Full and lightweight finetuning are evaluated as ways to reduce the performance gap between smaller and larger FMs.The comparison measures sample efficiency by labeled samples and training efficiency by parameter updates.
- Finetuning Methods: Full finetuning updates all model weights, whereas lightweight finetuning trains a nonlinear adapter on outputs from a frozen model.Adapters are generally less sample efficient because their trainable layer starts from scratch.
- Finetuning Methods: Lightweight finetuning recursively passes prompt representations through a frozen FM and adapter, updating only the adapter parameters.The second frozen-model pass produces the final answer.
- Finetuning Tradeoffs: Larger FMs can be used directly in zero- or few-shot settings, whereas smaller FMs require additional finetuning for similar performance.The figure summarizes the sample-training efficiency tradeoff.
- Finetuning Methods: Both finetuning approaches train models to generate task outputs from the paper’s natural-language prompts.Outputs include Yes/No strings or missing values, depending on the task.
- Experimental Goal: The experiments use GPT-3-1.3B and GPT-3-6.7B to study whether finetuning can close the gap with larger models.They specifically analyze sample-efficiency and training-efficiency tradeoffs.
A.3 Experimental Results
Finetuning can narrow the performance gap between smaller and larger FMs, with tradeoffs between sample efficiency, trainable parameters, and dataset-specific effectiveness. Adapter methods are cheaper and can match or exceed full finetuning in some settings, but may require more data.
- Full Finetuning: Full finetuning reduces the performance gap between GPT-3-6.7B and GPT-3-175B across datasets, using as little as 10% of Walmart-Amazon training data.GPT-3-1.3B matches GPT-3-175B on Walmart-Amazon and Restaurant, and comes within 8 points on Hospital.
- Lightweight Finetuning: Adapters bridge the GPT-3-6.7B-to-175B gap on Restaurant and Walmart-Amazon but not Hospital.For GPT-3-1.3B, the average performance difference from GPT-3-175B is 25 points across the three datasets.
- Sample Efficiency: Fully finetuned models are usually more sample efficient than adapters, while larger models achieve comparable performance with less training data.GPT-3-6.7B trained on 10% of the data equals or exceeds GPT-3-1.3B trained on 50% in both finetuning settings.
- Dataset-Specific Effects: The adapter setup may need more examples to learn generalizable patterns when the training set is particularly small.The Hospital training set contained 100 samples, where the 6.7B adapter performed poorly.
- Training Efficiency: GPT-3-6.7B adapters outperform full finetuning on 2 of 3 datasets using 5% as many trainable parameters.At GPT-3-1.3B, training efficiency trades off non-trivially against performance as model size decreases.
- Encoded Knowledge: Qualitative analyses show GPT-3-175B can use address-related functional dependencies to impute missing zip codes and cities.The analysis examines FM knowledge relevant to data tasks through qualitative and slice-based evaluations.
B.2 Slice analysis
The slice analysis examines how entity frequency affects imputation across training regimes. GPT-3-175B handles unseen entities, while finetuning is needed for rare entities and lightweight finetuning can outperform full finetuning there.
- Entity Frequency: GPT-3-175B is the only evaluated model that correctly imputes values for entities absent from the training set.The analysis uses city-name frequency counts to define three subclasses.
- Entity Frequency: Rare entities such as West LA require full or lightweight finetuning to learn their patterns.Lightweight finetuning is more accurate than full finetuning for rare subclasses with 100% and 50% of the training data.
C EFFECTS OF FM BIAS
FM pretraining can carry social biases into data-management outputs. In data repair, these biases may affect how entries are replaced or corrected, motivating mitigation and deployment monitoring.
- Bias Sources: FMs inherit biases from their large web-text pretraining corpora, including gender, religion, and race biases.These effects should be recognized when models are used in downstream applications.
- Deployment Risks: Biased FM outputs may cause data-repair systems to replace or correct entries in biased ways.The paper calls for mechanisms to mitigate model bias and monitor outputs during deployment.
D EVALUATING MORE FMS ON DATA WRANGLING TASKS
The paper evaluates GPT-3 on data integration and cleaning tasks and contributes those tasks to HELM for broader evaluation across large FMs.
- Scope and Benchmarking: The study evaluates GPT-3 on a collection of data integration and cleaning tasks.The tasks are contributed to HELM, which evaluates a broader set of large FMs on them.