PaperlayerView exact revision on arXiv

Source-linked AI summary

DIADA: Automatic Data Composition in Data Lakes

Marc Maynou, Albert Martin, Sergi Nadal, Anna Queralt, Oscar Romero

arXiv:2610.01646v1cs.DB

TL;DR

Data lakes make it difficult to organize attributes into meaningful relations: joinability does not establish relevance, and target-specific selection can discard attributes useful to other tasks. DIADA approximates multivariate dependence by mining predicate sets that violate an independence hypothesis. Experiments report that one target-agnostic composition improves downstream models across targets.

  • Problem

    Data composition asks how to organize fragmented data-lake attributes into meaningful relations without conditioning on a particular analytical target.

    Source: §1 · p. 1

  • Method

    DIADA maps attributes to predicates and mines their inclusion lattice for sets whose joint behavior departs from an independence hypothesis.

    Source: §3.2 · p. 5, §7 · p. 12

  • Results

    A single DIADA composition improves downstream models’ performance and generalization regardless of target, while filtering noisy attributes and preserving relevant ones.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Target-agnostic composition provides meaningful datasets that can serve diverse downstream modeling tasks without being conditioned on a particular target.

    Source: §1 · p. 3

    The scalable lattice heuristic may miss dependencies whose subsets carry no detectable signal, although its pairwise guarantee is unaffected.

    Source: §4.3 · p. 8

Abstract

from arXiv · show

Data lakes contain a plethora of attributes scattered across many tables that, when combined, provide enhanced assets for data analysis. Nonetheless, deciding which attributes belong together in meaningful relations remains a manual, per-task effort. Merging by joinability alone provides no guarantees regarding attribute relevance, while selecting features against a single target discards attributes useful to other tasks. To address this gap, we introduce the data composition problem: organizing a fragmented, heterogeneous lake into meaningful relations, agnostic of any particular analytical task so that the resulting organization can serve as a common foundation for diverse downstream analyses. We propose DIADA, a composition system that employs multivariate dependence as the criterion for assessing the meaningfulness of a relation and approximates it by hypothesizing independence among attributes and identifying those sets that violate this hypothesis. To do so, we map the attributes to a predicate space, forming a lattice under inclusion and mining those predicate sets that exhibit dependence among their constituents. We contribute a dedicated and scalable algorithm to effectively explore this space, outscaling classical algorithms for mining relationships, thus discovering dependencies that would otherwise be impractical to identify. We demonstrate that applying a single data composition process benefits diverse potential downstream tasks. This is the result of providing a subset of low-noise, statistically relevant attributes that increases the confidence that detected patterns are grounded in real relationships, thus preventing common modeling issues in large-scale environments.

1 INTRODUCTION

Data composition organizes fragmented data-lake attributes into meaningful relations for downstream analysis, rather than relying on noisy joins or target-specific selection. DIADA uses predicate-set independence testing to approximate multivariate dependence, and experiments report benefits across downstream tasks.

1 INTRODUCTION

  • Data composition organizes attributes from a fragmented lake into meaningful relations for downstream analysis, addressing a stage that is otherwise manual or automated with important limitations.Multiway joins can introduce unhelpful features; target-driven selection discards attributes useful to other tasks, while target-agnostic methods may preserve uninformative attributes.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • A meaningful relation is defined by multivariate dependence: attributes may interact jointly even when their individual marginal effects are weak.Fully estimating multivariate measures is exponential in the number of features, while existing approaches either fail to scale or focus on pairwise dependencies.Source: §1 · p. 2
  • DIADA maps attributes to predicates, forms an inclusion lattice, and mines predicate sets whose joint probabilities depart from the independence hypothesis.This provides an intermediate abstraction for dependency analysis across heterogeneous data.Source: §1 · pp. 2–3
  • DIADA’s dedicated algorithm uses an approximation factor to guide lattice pruning toward predicate sets whose relatedness remains uncertain.The approach provides a statistics-based criterion for meaningful relations and a scalable way to approximate multivariate dependencies.Source: §1 · p. 3
  • DIADA’s single, target-agnostic composition improves downstream models’ performance and generalization regardless of target, while filtering noisy attributes and preserving relevant ones.The experiments also report that DIADA outscales traditional miners and discovers dependencies that could otherwise be impractical to detect.Source: §1 · p. 3

2 RELATED WORK

Related work addresses pieces of data composition but does not provide a scalable, target-agnostic method for organizing complete relations according to multivariate relatedness. Existing approaches vary in what they preserve, assess, or can scale to.

2 RELATED WORK

  • The composition requirements are target-agnostic, multivariate, complete, and feasible at lake scale.Table 1 summarizes how the reviewed approach families satisfy these requirements.Source: §2 · p. 3, Table 1 · p. 3

2.1 Assembling data from a lake

  • Join and union discovery identify combinable tables, while lake-organization methods arrange assets by similarity; neither evaluates whether contributed attributes form meaningful relations.Supervised selection and augmentation are target-specific, so changing the analytical goal requires repeating the search and can discard attributes useful later.Source: §2.1 · p. 3

2.2 Unsupervised feature selection

  • Unsupervised feature selection preserves surrogate data structure, but correlated noise can be retained when the structure-preserving objective treats that structure as sound.It is the only reviewed family that satisfies, to some extent, all four composition requirements.Source: §2.2 · pp. 3–4

2.3 Discovering dependencies

  • Exact functional dependencies capture deterministic constraints, while statistical dependency methods such as CORDS assess correlations or soft dependencies; neither provides joint relatedness assessment over feature sets.Schema decomposition addresses the dual problem of splitting an existing relation and has exponential cost in the number of attributes, with reported empirical results up to 30 columns.Source: §2.3 · p. 4

2.4 Defining the composition criterion

  • Total correlation and related measures capture multivariate dependence but are exponential to estimate, while scalable alternatives often fall back on pairwise or greedy formulations.Predicate-based dependence search uses the itemset lattice but leverages independence rather than the frequency-based pruning principle of canonical pattern miners.Source: §2.4 · p. 4

3 A STATISTICAL CRITERION FOR MEANING

DIADA treats meaningful relations as sets of attributes that depart from independence, using a predicate abstraction to make statistical dependency analysis applicable to heterogeneous data. Data composition groups related attributes while retaining unrelated attributes and the keys needed to join the resulting relations.

3.1 Preliminaries

  • Exact multivariate dependence is infeasible in data lakes, so a useful criterion must account for sampling uncertainty, heterogeneous data, and objective association decisions.Source: §3.1 · p. 4

3.2 Predicate spaces

  • DIADA maps attributes to binary predicates over a domain, creating a common abstraction for dependency analysis across heterogeneous data types.The framework accommodates categorical and binned numerical attributes and allows predicates for complex types such as images or text.Source: §3.2 · pp. 4–5
  • Binary predicate projection loses information, so it is suited to detecting whether dependence exists but not to detailed characterization of dependencies.The implementation uses pair-of-tuples domains; non-tabular predicate examples are left to future work.Source: §3.2 · p. 5

3.3 Meaningfulness as a hypothesis test

  • Meaningfulness is tested against complete independence: predicate sets that depart from the independence hypothesis indicate relationships among their underlying attributes.A predicate-set violation guarantees attribute dependence, but the converse does not hold because the predicate abstraction can hide dependencies.Source: §3.3 · p. 5
  • Conditional invariance across every subset is equivalent to zero total correlation, connecting the test criterion to complete independence.Source: §3.3 · p. 5

3.4 Problem statement

  • Data composition takes the transitive closure of related-attribute pairs to form relations, retaining unrelated attributes separately and replicating needed join keys.A pair is related when it appears with another attribute in a predicate set that violates conditional independence.Source: §3.4 · p. 6

4 SCALABLE ALGORITHM

DIADA searches a predicate-set lattice using statistically supported independence violations to guide exploration rather than traversing the full space. Posterior refinement and pruning target the scalability challenges of large domains and exponentially many candidate sets, with a subset-traceability assumption bounding what the search can find.

4 SCALABLE ALGORITHM

  • Candidate predicate sets grow exponentially with attribute count, while domain evaluation creates a separate row-scalability challenge; arbitrary sampling or support cutoffs lack a validated general criterion.DIADA adapts these strategies with an approximation factor rather than arbitrary hyperparameters.Source: §4 · p. 6

4.1 Posterior modeling

  • DIADA models predicate probabilities with Beta posteriors from independently sized samples, accepting independence violations when their soundness reaches the statistical threshold.Posterior refinement continues until variance falls below approximation factor ε; Figure 2 illustrates weak pairwise signals guiding exploration to a strongly dependent triple.Source: §4.1 · p. 6, Figure 2 · p. 7

4.2 Distribution Refinement

  • Distribution refinement expands samples and updates posterior parameters without storing the samples, allocating effort in proportion to uncertainty until variances meet ε.New samples are propagated through the explored lattice by filtering along predicate edges.Source: §4.2 · p. 7

4.3 Lattice Expansion

  • Lattice expansion adds predicates from attributes already related to the explored set, pruning supersets whose attributes have shown no relationship with those present.This search assumes dependencies leave a detectable signal in a proper subset; purely interactive dependencies may be missed, although all cross-attribute pairs are explored.Source: §4.3 · pp. 7–8

4.4 Termination and cost

  • At termination, explored sets have uncertainty bounded by ε, and computational cost depends primarily on ε and the number of dependencies rather than exhaustive lattice traversal.Source: §4.4 · p. 8

5 THE DIADA SYSTEM

DIADA takes a wide relation, builds a predicate space for its attributes, mines statistically meaningful dependencies, and converts them into relations. Its composition combines compact tuple-pair predicates, independence testing, pairwise dependency scores, and graph components.

5 THE DIADA SYSTEM

  • DIADA constructs a predicate space for each input relation, mines statistically meaningful attribute dependencies, and transforms them into relations in the composed layer.Its input is typically a wide relation formed by joining a base relation with candidate datasets returned by a discovery system.Source: §5 · p. 8
  • Candidate datasets must be pre-aggregated by join attribute before a multiway left join when one-to-many matches would alter the sampled domain.Otherwise, match multiplicities reweight the empirical distribution, making resulting dependencies indistinguishable from genuine dependencies in the predicate space.Source: §5 · p. 8
  • Tuple-pair predicates use equality and inequality for every attribute, adding less-than and greater-than for ordered domains.This avoids enumerating distinct values, whose number makes tuple-level predicates impractical for large, heterogeneous datasets; the compact representation retains information needed for dependency discovery.Source: §5 · p. 8
  • DIADA infers joint predicate distributions with posterior variance bounded by user-specified ε, then identifies predicate sets violating conditional independence.Source: §5 · p. 8
  • For each attribute pair, the dependency score is the maximum soundness among conditional-probability pairs that violate the independence criterion.The score measures the greatest change in one attribute’s predicate behavior when conditioning on a predicate from the other attribute, and serves as the test statistic for the independence hypothesis.Source: §5 · p. 8
  • Pairwise scores form an undirected association graph, and each connected component becomes a relation in the composition.The cutoff uses the stated normal-distribution quantile with a Bonferroni correction; univariate noise is retained in a component, and columns used for joins are propagated into every resulting relation.Source: §5 · pp. 8–9

6 EXPERIMENTS

The experiments test DIADA’s scalability, signal/noise separation, modeling utility, and reuse across targets on 17 public datasets augmented with synthetic noise.

6 EXPERIMENTS

  • Evaluation used 17 public datasets converted into univariate- and multivariate-noise benchmarks, with injected noise providing known columns to remove.Validation on naturally occurring heterogeneous lakes remains future work; uncontrolled compositions could combine all datasets into a single component.Source: §6 · p. 9

6.1 Scalability of the mining algorithm

  • On rows, DIADA matched FP-Growth while avoiding global compression structures; for columns, it expanded the lattice only when statistical evidence warranted it.Apriori and FP-Growth exhaustively enumerate the lattice, whereas DIADA’s evidence-guided exploration reduces unnecessary searches.Source: §6.1 · pp. 9–10, Figure 4 · p. 10

6.2 Separating signal from noise

  • DIADA selected 1.6% noise columns while retaining 91.5% original attributes, and it was the only method that never discarded prediction targets.The comparison averaged percentages of noise columns, base columns, and targets kept across benchmarks.Source: Table 4 · p. 10, §6.2 · p. 10

6.3 Improving downstream pipelines

  • Post-composition utility never fell below pre-composition and was often considerably better, while against base it was generally equal or sometimes higher.Without composition, several difficult-target benchmarks selected up to 60% noisy features; post-composition models used almost exclusively original features.Source: Figure 5 · p. 11, §6.3 · p. 11

6.4 One composition, any target

  • A single composition performed close to base and better than pre-composition models for both synthetic targets across most benchmarks, without seeing either target.Noise was almost eradicated from the resulting relations; the only noticeable exception was the second target of jannis.Source: Figure 6 · p. 12, §6.4 · p. 12

7 CONCLUSION

The conclusion frames data composition as organizing fragmented, heterogeneous data lakes into meaningful relations without conditioning on specific targets. DIADA approximates multivariate dependence over predicate sets to produce scalable, target-agnostic compositions for diverse downstream tasks.

7 CONCLUSION

  • Data composition organizes fragmented, heterogeneous data lakes into meaningful relations without conditioning on specific targets.Source: §7 · p. 12
  • DIADA approximates multivariate dependence by mining dependent predicate sets in an inclusion lattice with a dedicated, scalable algorithm.The algorithm scales beyond classical mining algorithms, enabling detection of dependencies that would otherwise be impractical.Source: §7 · p. 12
  • Its target-agnostic compositions provide low-noise, statistically meaningful attributes for diverse downstream tasks.Source: §7 · p. 12
  • Future work targets information loss from predicate abstraction, safeguards against single-component composition, and extensions to heterogeneous domains and other predicate spaces.Source: §7 · p. 12
Loading 2610.01646v1…