Source-linked AI summary
Discovering Relationships in Data Lakes Using Large Language Models: An Industrial Case
Ahlame Diouan, Eric Ferey, Sabine Loudcher, Jérôme Darmont
TL;DR
Data lakes are difficult to use when weak metadata and coded ERP schemata obscure relationships between columns. ColRel combines ingestion-time evidence with optional dictionary-grounded LLM descriptions in a two-stage embedding-retrieval method. Experiments report clear gains from semantic enrichment in challenging semantically related settings where lexical overlap is weak and equality joins fail.
Problem
Weak or missing metadata, fragmented ERP tables, and coded or abbreviated column names make column relationship discovery difficult and hinder scalable data-lake exploration.
Method
ColRel embeds ingestion-time schema and value evidence, then optionally enriches column representations with business-dictionary-grounded LLM descriptions before retrieval.
Results
In semantically related, weak-signal settings, ColRel–Stage2 provides clear ranking-quality and shortlist-coherence gains without changing the embedding and similarity computation pipeline.
Takeaways & Limitations
ColRel is particularly effective for discovering semantically related columns when heterogeneous conventions, coded identifiers, and weak lexical overlap limit direct value-based matching.
Takeaways & Limitations
The evaluation focuses on ERP data, and adapting ColRel to a new domain mainly requires an appropriate user- or expert-provided dictionary or equivalent grounding resource.
Abstract
from arXiv · showhide
Data lakes rely on metadata to remain usable, yet this meta data is often limited or weakly informative for column relationship discovery, especially in ERP-derived datasets with coded or abbreviated schema labels. We propose ColRel, a two-stage method that builds column embeddings from metadata and data available at ingestion time. In difficult cases, such as coded schemata, business dictionaries help better interpret column names and support the generation of short natural-language descriptions used in the second stage. Experiments on public benchmarks and an industrial ERP dataset show that ColRel is particularly effective in semantically related, weak-signal settings.
1 Introduction
Data lakes become difficult to explore when metadata is missing or weak, especially for ERP data with fragmented tables and coded column names. ColRel addresses this challenge with two-stage embedding retrieval and optional LLM-based semantic enrichment.
- Weak metadata makes discovering relationships between heterogeneous data-lake tables largely manual, particularly in ERP-derived datasets with abbreviated or coded column names.
- ColRel is a two-stage method for discovering column relationships from ingestion-time schema cues, representative values, and optional semantic descriptions.
- Stage 2 uses an LLM to generate concise descriptions when metadata signals are weak, then injects them into the embedding input.
- Columns are encoded with a sentence embedding model, compared using cosine similarity, and returned as top-ranked candidates for validation.
- The work evaluates ColRel against established schema-matching baselines on public datasets and an industrial public-housing ERP dataset, including a synthetic counterpart.
2 Related Work
Schema matching and data-lake discovery increasingly use representation learning and language models, but coded names and weak metadata remain difficult. ColRel targets this gap by retrieving column relationships in weak-metadata ERP environments with optional dictionary-grounded semantic enrichment.
- Classical schema matchers combine schema and instance information but degrade when data-lake metadata are missing and column names are coded.
- Data-lake discovery systems retrieve related tables and suggest joins, with Aurum supporting enterprise exploration and LakeBench evaluating joinability and unionability.
- Representation-based approaches encode schema elements into dense vectors and retrieve candidates through nearest-neighbor search.EmbDI learns from relational structures, while DeepJoin uses pre-trained language models and column evidence for join discovery.
- Recent methods use LLMs as schema matchers or enrich columns with semantic tags before matching.Examples include direct prompting, retrieval-augmented ReMatch, and SMUTF’s generated semantic tags.
- ColRel differs by focusing on column relationship discovery rather than strict schema alignment in weak-metadata ERP environments.Its optional enrichment stage generates concise, dictionary-grounded column descriptions and injects them into embedding inputs.
3 ColRel: Column Relationship Discovery
ColRel discovers ranked column correspondences from ingestion-time metadata and data through a two-stage embedding pipeline. It addresses ERP data with coded schema labels by optionally generating dictionary-grounded descriptions, then retrieves candidates for human validation.
- ColRel targets meaningful column relationships in heterogeneous ERP data lakes with coded or abbreviated names and weak metadata.The goal is to identify related column pairs even when lexical or structural signals are limited.
- The method uses column names, data types, values, and intra-metadata available at ingestion time to rank related column pairs by confidence.
- Joinable columns represent the same concept and have sufficiently overlapping values for an equality join, whereas semantically related columns may lack verbatim value equality.The latter can reflect formatting differences, noise, or heterogeneous conventions.
- Dictionary entries can turn opaque tokens such as dt and construct into a description like “construction date of the asset” before embedding.This enriches weak Stage 1 evidence and brings the representation closer to a semantically equivalent column.
- ColRel represents each column with a single embedding built from ingestion-time evidence and, when enabled, an LLM-generated description.The overall workflow includes text construction, description generation, encoding, similarity retrieval, and human validation.
- Column-to-text representations combine raw and tokenized names, primitive types, and a fixed-size sample of representative values.
- An LLM generates concise column descriptions from the column representation, augmented with business-dictionary mappings, domain hints, and construction rules when available.
- The final column input concatenates ingestion-time text with the generated description when available, preserving one vector per column while injecting semantic cues.
4 Experimental Setup
The evaluation compares classical matchers with ColRel’s two embedding stages across joinable and semantically related discovery tasks. It uses public benchmarks and a synthetic industrial ERP-inspired dataset, with ranked retrieval as the primary practical objective.
- Evaluation design: The study compares classical matchers, ColRel–Stage 1 embeddings, and ColRel–Stage 2 semantic enrichment plus embeddings.Stage 1 and Stage 2 share the same encoder and similarity pipeline, isolating the incremental effect of generated descriptions.
- Metrics: Evaluation emphasizes ranked candidate retrieval using Hit@5 and MRR, while one-to-one precision, recall, and F1 support comparison with Valentine.Hit@5 reflects whether a correct match appears within a five-candidate human validation budget.
- Datasets: Experiments cover ChEMBL clean and noisy benchmarks plus a synthetic Public Housing dataset preserving industrial ERP schema characteristics.Ground truth comes from Valentine for ChEMBL and a domain expert for Public Housing.
- ColRel representations: ColRel encodes ingestion-time column names, types, tokenized variants, and representative values into dense vectors for nearest-neighbor retrieval.The evaluated encoders span compact MiniLM models, general-purpose MPNet, and retrieval-oriented BGE.
- Implementation: The implementation uses a single Nvidia RTX A6000 GPU and compact representations to balance encoder efficiency, runtime, and scalability.Model selection is guided by MTEB to cover an efficiency–quality spectrum.
5 Results and Discussion
Joinable cases are generally strong because shared identifiers and value overlap provide reliable evidence, whereas semantically related cases expose weak lexical and instance signals. ColRel–Stage 2 is most beneficial in noisy, coded settings, especially for Public Housing shortlist quality.
- Joinable results: Joinable cases produce near-ceiling Hit@5 and MRR because shared identifiers and value overlap make correct matches easy to retrieve.Differences mainly concern ranking consistency, robustness to schema noise, and runtime.
- Joinable results: On ChEMBL–Noisy, BGE-m3 achieves Rec@|GT| / Hit@5 / MRR of 0.79 / 1.00 / 0.97, while MPNet and MiniLM-L12 drop to Rec@|GT| values of 0.28 and 0.26.Their Hit@5 remains 0.89–0.93, indicating retrieval with less consistent top ranking.
- Semantically related results: Semantically related cases are harder because heterogeneous conventions, coded identifiers, and weak lexical overlap make instance- and string-based signals insufficient.Ranking consistency becomes the main difficulty in this regime.
- Semantically related results: On ChEMBL–Clean, ColRel–Stage 2 raises Rec@|GT| to 0.90 for MiniLM-L12 and MPNet, with MiniLM-L12 reaching MRR = 1.00.The gains are moderate because Stage 1 already performs strongly.
- Discussion: Documentation-based dictionary grounding improves robustness in coded ChEMBL schemata, motivating expert-validated business dictionaries for ERP settings.The ChEMBL dictionary was manually constructed without expert assistance and was not validated by domain experts.
- Semantically related results: On Public Housing, ColRel–Stage 2 BGE-m3 reaches Hit@5 / MRR of 0.82 / 0.68, versus 0.53 / 0.53 for the best classical shortlist.Other Stage 2 variants remain stronger than Stage 1, with Hit@5 = 0.71 and MRR between 0.51 and 0.57.
6 Conclusion and Future Work
ColRel combines embedding-based retrieval with optional business-dictionary and LLM-based semantic enrichment for column relationship discovery in weak-metadata ERP data. The paper reports gains in semantically related, weak-signal settings and identifies domain grounding, scalability, privacy, and interactive refinement as future directions.
- Conclusion: ColRel performs embedding-based retrieval and optionally enriches column representations with business dictionary and LLM-generated descriptions.The method targets ERP-derived data characterized by coded schemata and weak metadata.
- Conclusion: ColRel–Stage2 improves ranking quality and shortlist coherence when lexical overlap is weak and equality joins fail.The gains come from grounded semantic enrichment without changing the embedding and similarity computation pipeline.
- Scope: ColRel’s adaptation to new domains mainly requires an appropriate user- or expert-provided dictionary, terminology resource, or equivalent grounding support.The evaluation focuses on ERP data, although the representation approach can apply elsewhere when schema cues, representative values, and business dictionaries are available.
- Future Work: Future work includes interactive feedback, improved grounding, scalability through indexing and incremental updates, and investigation of privacy and confidentiality issues.The authors also point to locally deployable language models as an alternative to reduce exposure to external services.
- Future Work: Generated semantic descriptions could support broader metadata enrichment, including data cataloging and governance within industrial data lakes.