PaperlayerView exact revision on arXiv

Source-linked AI summary

Data Lakes: A Survey of Functions and Systems

Rihan Hai, Christos Koutras, Christoph Quix, Matthias Jarke

arXiv:2106.09592v2cs.DB

TL;DR

Data-lake research lacks a coherent account of its problems and solutions. This survey reviews the field’s development, architecture, and systems, classifying approaches by function. It finds that storage choices are diverse and shape the functions a data lake requires.

  • Problem

    Data-lake definitions and functions are ambiguous, and a coherent picture of their problems and solutions is missing.

    Source: §1.1 · p. 1

  • Method

    The survey reviews data-lake development, architecture, and systems, using a three-level categorization to analyze methods by function.

    Source: §9 · p. 17

  • Results

    Storage choices span file systems or databases, single or hybrid systems, and on-premise or cloud, and the strategy often shapes required functions.

    Source: §4.5 · p. 7

  • Takeaways & Limitations

    Data-lake designers need to consider both raw data and its intended use when choosing a storage system.

    Source: §4.5 · p. 7

    Data lakes lack the predefined join dependencies or functional dependencies used in relational databases, constraining direct transfer of those workflow-optimization approaches.

    Source: §8.2 · p. 17

Abstract

from arXiv · show

Data lakes are becoming increasingly prevalent for big data management and data analytics. In contrast to traditional 'schema-on-write' approaches such as data warehouses, data lakes are repositories storing raw data in its original formats and providing a common access interface. Despite the strong interest raised from both academia and industry, there is a large body of ambiguity regarding the definition, functions and available technologies for data lakes. A complete, coherent picture of data lake challenges and solutions is still missing. This survey reviews the development, architectures, and systems of data lakes. We provide a comprehensive overview of research questions for designing and building data lakes. We classify the existing approaches and systems based on their provided functions for data lakes, which makes this survey a useful technical reference for designing, implementing and deploying data lakes. We hope that the thorough comparison of existing solutions and the discussion of open research challenges in this survey will motivate the future development of data lake research and practice.

1 INTRODUCTION

Data lakes address the difficulty of managing diverse data across silos, but their definition, functions, and solutions remain fragmented. This survey reviews the field and organizes systems by function to clarify data-lake design and research.

1 INTRODUCTION

  • Varied, fast-growing data from sources such as sensors, transactions, and social media is often siloed, making integration important for deriving insights.The variety of data remains a particular challenge, with open issues in its collection, storage, and processing.Source: §1 · p. 1
  • Data lakes address limits of schema-on-write by storing heterogeneous raw data in original formats for on-the-fly maintenance, querying, and analytics supported by metadata.The survey contrasts this approach with warehouse ETL and schema-on-read data handling.Source: §1 · p. 1

1.1 Survey goal and related work

  • Despite many proposals, data-lake definitions and functions remain ambiguous, and a coherent picture of problems and solutions is missing.The survey aims to help researchers and developers build or customize data lakes and identify open questions and future directions.Source: §1.1 · p. 1
  • The survey distinguishes itself from prior work through a broader functional view and a finer-grained comparison of existing data-lake systems.It connects conceptual architecture to system functions and covers topics beyond architecture and metadata management.Source: §1.1 · pp. 1–2

1.2 Contributions and outline

  • The survey reviews the field’s development, presents a fine-grained architecture and function-based classification, and compares existing approaches.Its scope centers on systems that explicitly claim to be data lakes or provide partial data-lake functions; the survey is organized from fundamentals and storage through functional tiers to future directions.Source: §1.2 · p. 2

2 A BRIEF HISTORY OF DATA LAKES

The history of data lakes moves from an industry concept for retaining raw data to architectures shaped by concerns about metadata and governance. Since 2016, proposals, systems, and research have diversified.

2 A BRIEF HISTORY OF DATA LAKES

  • The concept evolved over roughly a decade, with the survey organizing its history into three stages.The stages are beginnings, criticism and further development, and subsequent prosperity and diversity.Source: §2 · p. 2

2.1 2010–2013: Beginnings

  • Early proposals framed data lakes as a way to retain raw data and support diverse needs, while Pivot proposed ingestion, insight, and action tiers in 2013.Pivot gave few details about actual implementation.Source: §2.1 · pp. 2–3

2.2 2014–2015: Criticisms and further development

  • Criticism that raw, disparate data could become an unusable “data swamp” emphasized metadata management, governance, provenance, security, and privacy.Later proposals added functions including metadata extraction and enrichment, schema-on-read, AI, and crowdsourcing.Source: §2.2 · p. 3

2.3 2016–present: Prosperity and diversity

  • Since 2016, data-lake work has diversified across architectures, commercial and open-source tools, and research systems addressing specific challenges.The surveyed research challenges include metadata management, quality, provenance, enrichment, organization, modeling, integration, and dataset discovery.Source: §2.3 · p. 3

3 DATA LAKE ARCHITECTURE AND PROPOSED

The section proposes a function-oriented architecture and classification for data-lake systems, addressing the lack of technical detail in broad architectural philosophies. It also situates lake-specific challenges and human users within that framework.

3.1 Pond and zone architectures

  • Pond architecture moves data among ponds according to status and use, whereas zone architecture separates a dataset's lifecycle into stages such as loading, validation, discovery, and analysis.Source: §3.1 · p. 4

3.2 Proposed architecture and categorization criteria

  • The proposed architecture specifies functions by tier to clarify the data-lake workflow and enable fine-grained comparison of existing systems.Earlier high-level philosophies often lack technical detail that can hamper modular and reusable implementations.Source: §3.2 · p. 4
  • Ingestion, maintenance, and exploration tiers link data-lake users to storage, covering import, dataset organization and discovery, and user access.Visualization and machine learning can sit in external application tiers above them.Source: §3.2 · p. 4
  • The survey classifies solutions by tiers (when), functions (what), and methods (how), allowing systems with multiple functions to appear in more than one category.The classification focuses on existing data-lake systems and functions identified in data-lake-related studies.Source: §3.2 · p. 4
  • Because raw data may lack a usable schema and metadata may not all be available at ingestion, discovery and enrichment can continue during maintenance or querying.The section places ongoing cleaning and provenance activities in maintenance.Source: §3.2 · p. 4

3.3 Data lake users

  • Data-lake users include analysts, curators, governance teams, and operations staff, but the survey focuses on technical system comparison rather than human-system interaction.Users may contribute metadata tags and linkage information, while diverse users and stores also raise security-control challenges.Source: §3.3 · pp. 4–5

4 STORAGE

The survey compares data-lake storage options across files, single databases, polystores, and cloud platforms. It emphasizes that storage choices should reflect both raw data and intended use because they shape required functions.

4 STORAGE

  • Data-lake storage choices include files, single relational or NoSQL databases, polystores, and cloud or on-premise deployments.The survey classifies solutions by whether ingested data is stored as files, in a single database, or using polystores.Source: §4 · p. 5

4.1 File-based storage systems

  • HDFS supports diverse file formats and is widely used, but Hadoop alone usually does not fulfill data-lake goals; Azure Data Lake Store and CLAMS add capabilities.Azure combines hierarchical multi-tier file storage with Blob storage and Hadoop ecosystem support; CLAMS supports dataset registration for constraint discovery and cleaning.Source: §4.1 · p. 5

4.2 Single data store

  • A single database can suit specific data-lake settings: the personal data lake stores heterogeneous personal-data fragments as extensible-metadata property graphs in Neo4j.The described application focuses on user data, which is usually smaller than business data and has higher privacy requirements.Source: §4.2 · p. 5

4.3 Polystore systems

  • Polystores integrate multiple data stores for heterogeneous raw data; examples distribute data across relational, document, graph, and file systems.Constance selects stores by format and can use HDFS for cases such as scalable processing of large binary-image streams; users may override defaults.Source: §4.3 · p. 6

4.4 Cloud data lakes

  • Cloud data lakes commonly use single-cloud, multi-cloud, or hybrid deployments, offering dynamic scaling and analytics tools while posing security, provenance, and fault-tolerance challenges.Source: §4.4 · p. 7

4.5 Summary and discussion

  • Storage selection should account for both raw data and intended use because the diverse storage strategy often shapes the functions a data lake requires.The survey identifies further research on data-lake solutions built over cloud platforms as an open direction.Source: §4.5 · p. 7

5 INGESTION

The ingestion section treats metadata acquisition and modeling as necessary for making raw data usable. It reviews extraction approaches and metadata models, noting that graph-based models appear more flexible for data-lake needs.

5 INGESTION

  • Raw data without metadata may be difficult to use, so ingestion solutions extract and model metadata such as schemas, semantics, constraints, and descriptive information.Source: §5 · p. 7

5.1 Metadata extraction

  • Metadata extraction discovers dataset metadata, often structural information first, as well as semantic information and relationships to other datasets.GEMMS parses heterogeneous input formats for structural metadata; DATAMARAN extracts log structures without human supervision, and Skluma targets content and context metadata in scientific files.Source: §5.1 · p. 7

5.2 Metadata modeling

  • Metadata modeling organizes metadata to support findability, accessibility, interoperability, and reuse; the survey groups models as generic, data-vault, or graph-based.Source: §5.2 · p. 8

5.2.1 Generic metadata models

  • Generic models separate content, semantic, and structural metadata, while HANDLE represents data, metadata, and properties and can adapt to zone architectures.GEMMS represents general properties as key-value pairs and structural metadata as trees and matrices; HANDLE can be implemented in Neo4j.Source: §5.2.1 · p. 8

5.2.2 Data vault

  • Data vault models hubs for business concepts, links for many-to-many relations, and satellites for descriptive properties, but reported applications note source inconsistencies.The cited work describes use in manufacturing, finance, and customer service.Source: §5.2.2 · p. 8

5.2.3 Graph-based metadata models

  • Graph-based metadata models represent datasets and their relationships as graphs or hypergraphs, with proposals supporting semantic links, dataset discovery, and metadata evolution.Aurum's enterprise knowledge graph is maintained as data changes and queried through discovery primitives.Source: §5.2.3 · p. 8

5.2.4 Summary and discussion

  • The survey finds data-vault models less suited to unstructured data and flexibility, while graph-based models appear more promising for flexibility and knowledge-graph linking.Structured metamodels can be extensible, but managing metadata storage and user interfaces remains challenging.Source: §5.2.4 · p. 8

6 MAINTENANCE

The maintenance tier groups functions that make heterogeneous raw data usable, including dataset organization. The survey compares organization systems by their structures, methods, and differing use cases.

6 MAINTENANCE

  • Maintenance functions address the work needed after heterogeneous raw data is ingested, including finding metadata, discovering relationships, and integrating, transforming, or cleaning data.The survey groups maintenance-related functions into seven categories.Source: §6 · p. 8

6.1 Dataset organization

  • Dataset organization structures and navigates heterogeneous collections, using catalogs, classification models, or directed acyclic graphs (DAGs) for distinct goals.KAYAK, and Juneau serve data science; DS-Prox,, is a pre-filtering step for schema matching.Source: §6.1 · p. 9

6.1.1 Catalog-based organization

  • GOODS, catalogs datasets with metadata collected after dataset creation, organized into categories tailored to Google's information-retrieval needs.The survey notes that developers applying catalog-based organization should customize the catalog to their own needs.Source: §6.1.1 · p. 9

6.1.2 Classification model based organization

  • DS-kNN extracts data and metadata features, uses Levenshtein distance to find similar datasets, then assigns a new dataset to the most frequent category among its top-k neighbors.If no existing datasets are found, it assigns the dataset to a new category; later work uses supervised ensemble models to estimate pairwise similarity.Source: §6.1.2 · p. 9

6.1.3 DAG-based organization

  • DAG-based organization serves several roles: KAYAK, models pipeline and task dependencies, Nargesian et al. model attribute containment, and Juneau models notebook workflows and variable dependencies.KAYAK's task-dependency DAG identifies tasks that can run in parallel; Juneau uses subgraph isomorphism to find tables with similar notebook workflows.Source: §6.1.3 · pp. 9–10

6.1.4 Summary and discussion

  • The approaches have different limits: GOODS, is tailored to Google's dataset practices, while some DAG methods rely on attribute containment and do not support fuzzy similarities.KAYAK, computes inter-dataset metadata for equi-joins among tabular datasets; Juneau uses workbook metadata as a similarity signal.Source: §6.1.4 · p. 10

6.2 Related dataset discovery

Related dataset discovery finds relevant data without querying or integrating every dataset in a lake. The surveyed systems use different relatedness signals and techniques, with approaches spanning joinability, data-science tasks, semantics, and large-scale discovery.

6.2 Related dataset discovery

  • Related dataset discovery narrows a large lake to datasets relevant to a purpose, using signals such as similar attribute names or overlapping values.Discovered relationships also provide metadata useful for exploring a lake and for tasks such as entity resolution or resolving inconsistencies.Source: §6.2 · p. 10

6.2.1 Discovery of joinable datasets

  • Joinable-dataset systems use varied methods: Aurum indexes column signatures; JOSIE [155] performs exact top-k overlap search with inverted indexes and a cost model; D3L combines five similarity signals.Brackenbury et al. also include humans when algorithms cannot provide reliable suggestions.Source: §6.2.1 · pp. 10–11

6.2.2 Discovery of task-specific datasets for data science

  • Juneau,, ranks tables for data-science tasks using task-relevant signals, including workflow provenance, matched attributes, and descriptive metadata.For data cleaning, it considers value overlap, schema overlap, provenance similarity, and differences in null values.Source: §6.2.2 · pp. 11–12

6.2.3 Discovery of semantically related datasets

  • PEXESO [40] compares vectors of textual values to find semantically joinable tables, while RNLIM uses language-model representations to classify semantic relationships between attributes.PEXESO [40] uses an inverted index and hierarchical grid; RNLIM draws on table and attribute names, data types, and value domains.Source: §6.2.3 · p. 12

6.2.4 Scalable related dataset discovery

  • DLN targets related-column discovery in enterprise lakes with petabytes or exabytes of streaming data, using random-forest classifiers trained from query JOIN clauses and sampled negative pairs.Its classifiers use metadata features, with data-based features also included for textual attributes in the ensemble model.Source: §6.2.4 · p. 12

6.2.5 Summary and discussion

  • Across the surveyed approaches, discovery typically extracts relatedness signals, computes and aggregates similarities, and often uses LSH-based indexing; systems differ in focus, from workflow tasks to exabyte-scale lakes.The survey notes that discovery criteria, similarity measures, and techniques vary significantly, and that categories can overlap.Source: §6.2.5 · p. 12

6.3 Data integration

Data integration combines heterogeneous sources and provides unified access, often after relevant datasets have been discovered. The surveyed systems include a multi-stage integration pipeline and an approach for integrating related tables discovered in a lake.

6.3 Data integration

  • Data integration combines heterogeneous sources for unified user access, and may follow discovery of relevant datasets before resolving differences in data models and schemata.The survey identifies schema matching, schema mapping, entity linkage, and query reformulation as fundamental integration steps.Source: §6.3 · pp. 12–13
  • Constance performs schema matching and partial integration, then maps schemas, rewrites and executes queries across stores, and resolves type and value conflicts in combined results.It also pushes selection predicates to source systems to reduce the data loaded.Source: §6.3 · p. 13
  • ALITE integrates related tables returned by unionable and joinable searches by embedding and clustering columns, then computing an optimized Full Disjunction.Its column embeddings use language-model techniques such as TURL.Source: §6.3 · p. 13

6.4 Metadata enrichment

Metadata enrichment creates implicit information from raw data to help users understand and explore datasets, although it can be computationally costly or require human effort. The survey distinguishes semantic, structural, and descriptive enrichment.

6.4 Metadata enrichment

  • Metadata enrichment creates implicit metadata from raw data, including hidden information that supports understanding and exploring datasets in a lake.The process may be costly or impossible or unnecessary during ingestion; enriched metadata can support provenance, discovery, cleaning, and organization.Source: §6.4 · p. 13

6.4.1 Semantic metadata enrichment

  • Semantic enrichment systems infer hidden meanings: CoreDB, adds synonyms and stems to extracted features, while D4 and DomainNet [85] discover domains and address ambiguous values.D4 represents domains with sets of terms; DomainNet [85] builds a graph from values and attribute names, then applies community detection.Source: §6.4.1 · p. 13

6.4.2 Structural metadata enrichment

  • Constance enriches structural metadata by finding relaxed functional dependencies that can hold despite inconsistent tuples or similar attribute values.These dependencies indicate that attributes may depend on others in a loose manner.Source: §6.4.2 · p. 13

6.4.3 Descriptive metadata enrichment

  • Descriptive enrichment can involve human experts: GOODS, uses crowdsourcing to add dataset descriptions and flag datasets for additional security attention.Participants can include data owners, auditors, and users from different teams.Source: §6.4.3 · pp. 13–14

6.5 Data cleaning

Data cleaning in data lakes addresses quality problems in ingested raw data, ideally across heterogeneous data with reduced human effort. The survey groups approaches by constraint inference and validation-rule inference.

6.5 Data cleaning

  • Data cleaning in lakes targets quality issues in ingested raw data, where volume and variety make heterogeneous-data handling and reduced human effort desirable.The survey distinguishes constraint inference from validation-rule inference.Source: §6.5 · p. 14

6.5.1 Data cleaning by constraint inference

  • CLAMS infers constraints from RDF data, ranks triples that violate them using a hypergraph, and asks users to validate removal candidates.The hypergraph records how many constraints each triple violates.Source: §6.5.1 · p. 14

6.5.2 Data cleaning by validation rule inference

  • Constance uses relaxed functional dependencies to identify potentially erroneous data objects, while Song et al. [138] infer validation rules for changing machine-generated string data.The validation-rule optimization balances false-positive-rate minimization against preserving quality issues.Source: §6.5.1 · p. 14, §6.5.2 · p. 14

6.6 Schema evolution

Data lakes face schema evolution amid heterogeneous schemas and frequent data and metadata changes. The survey also reviews provenance as metadata about data origins and use, including graph-based representations.

6.6 Schema evolution

  • Data-lake schema evolution must accommodate heterogeneous schemas and frequent changes to data and metadata, unlike relational warehouse schemas that are usually updated infrequently.The survey notes that data lakes are more agile systems.Source: §6.6 · p. 14
  • Klettke et al. reconstruct NoSQL entity-type histories by timestamping structures, detecting operations between versions, and using user validation for alternatives.Their approach also detects k-ary inclusion dependencies under an assumption of less-normalized NoSQL schemas.Source: §6.6 · p. 14

6.7 Data provenance

  • Provenance records data origin, usage, and lifecycle status; surveyed systems represent it in graphs for tracking transformations and answering relationship queries.GOODS visualizes provenance paths, CoreDB uses DAGs, and Juneau links notebook variables through shared functions.Source: §6.7 · pp. 14–15

7 EXPLORATION

Data-lake exploration addresses discovery across many heterogeneous sources, either by finding related datasets or by providing unified querying interfaces. The survey describes query-driven search and systems for querying heterogeneous data.

7 EXPLORATION

  • Exploration is challenging because lakes contain many heterogeneous sources and users rarely know all datasets; surveyed solutions pursue related-dataset discovery or unified querying.Source: §7 · p. 15

7.1 Query-driven data discovery

  • Query-driven discovery returns top-k related tables from a user-specified table, with search variants for column relatedness, attribute coverage, or an external task.Systems rank candidates using similarity estimates and indexes; D3L can also add non-top-k tables that improve attribute coverage when joined with top-k results.Source: §7.1 · p. 15
  • Existing discovery work mainly evaluates similarity accuracy or query time; deeper analysis of top-k result accuracy and completeness is rare, and most systems focus on tabular data.The survey presents exploration as a search problem that can precede data integration or data-science tasks.Source: §7.1 · p. 15

7.2 Heterogeneous data querying

  • Heterogeneous-query systems expose unified access through REST, SQL, or SPARQL interfaces, using source-specific query engines or mappings to retrieve and combine data.Ontario decomposes SPARQL queries using dataset profiles, while Squerall maps diverse sources to a mediator and joins and transforms retrieved entities.Source: §7.2 · pp. 15–16

8 CHALLENGES AND FUTURE DIRECTIONS

The survey identifies organizational and technical challenges for data lakes, including support for digital transformation, machine learning, and advanced analytics. It outlines open directions spanning ML-aware lake design and Lakehouse capabilities.

8.1 Data lakes in digital business transformation

  • Data lakes are proposed as mediating stores between changing business data streams and evolving analytics tasks, but realizing this organizational pattern is not trivial.The pattern envisions adding a newly acquired company's raw data to the lake and linking it with existing content.Source: §8.1 · p. 16
  • Current data lakes lack mature functions for machine learning and traditional analytical management, making them less adequate for complex industrial analytical workloads.The survey highlights transaction management, indexing, and caching as missing traditional functions.Source: §8.1 · p. 16

8.2 Data lakes meet machine learning

  • ML-aware lakes need abstractions and representations for multimodal training data, lake-specific ML preparation and lifecycle integration, and metadata methods covering models and their datasets.The survey raises discovery and cleaning of raw heterogeneous datasets and optimization of the combined data-management and ML lifecycle as research questions.Source: §8.2 · pp. 16–17
  • ML workflow optimization in lakes must use fuzzy, discovered metadata such as probabilistic joinability or relaxed dependencies rather than relying on predefined database metadata.The stated goal is improving model effectiveness and efficiency, including accuracy and training time.Source: §8.2 · p. 17

8.3 Advanced analytics and transaction management

  • Lakehouses combine data lakes' large-scale raw-data storage over open formats with data-warehouse analytics capabilities, including transaction management, indexing, caching, and metadata management.The survey identifies cloud-native read-write storage guarantees and auxiliary indexes over open formats as open challenges.Source: §8.3 · p. 17

9 CONCLUSION AND OUTLOOK

The survey looks back at data lakes’ first decade and reviews architectures, storage systems, and methods organized by function in a three-level categorization. It identifies future challenges and technologies, including machine learning and cloud computing, and forecasts rapid growth in data lake applications.

9 CONCLUSION AND OUTLOOK

  • The survey reviews data lake methods by function and uses a three-level categorization to analyze their corresponding research questions.It also presents a fine-grained architecture and discusses storage systems.Source: §9 · p. 17
  • The survey looks back at data lakes’ origin and development during their first decade of growing interest from academia and industry.Source: §9 · p. 17
  • The authors identify research and engineering challenges involving machine learning and cloud computing, and foresee explosive development of data lake applications.They describe the field’s golden age as yet to come.Source: §9 · p. 17
Loading 2106.09592v2…