Source-linked AI summary
On data lake architectures and metadata management
Pegdwendé Sawadogo, Jérôme Darmont
TL;DR
Big data challenges traditional management systems, while the data lake concept remains ambiguous and is sometimes confused with Hadoop. This survey develops a comprehensive state of the art covering definitions, architectures, metadata management, and trade-offs, while identifying limitations and open design issues.
Problem
Big data exceeds traditional systems, and data lake concepts and design approaches remain ambiguous, motivating a comprehensive review.
Method
The paper synthesizes data lake definitions, architectures, technologies, metadata classifications and systems, and design alternatives.
Results
The survey proposes a data lake definition, a Functional × Maturity architecture typology, metadata-system features, and a comparison of eighteen metadata systems and models.
Takeaways & Limitations
The paper frames metadata management as necessary to avoid data swamps and identifies architecture and metadata design choices for building data lakes.
Takeaways & Limitations
The proposed hybrid architecture remains incomplete because existing approaches may delete raw data or restrict access to refined data.
Abstract
from arXiv · showhide
Over the past two decades, we have witnessed an exponential increase of data production in the world. So-called big data generally come from transactional systems, and even more so from the Internet of Things and social media. They are mainly characterized by volume, velocity, variety and veracity issues. Big data-related issues strongly challenge traditional data management and analysis systems. The concept of data lake was introduced to address them. A data lake is a large, raw data repository that stores and manages all company data bearing any format. However, the data lake concept remains ambiguous or fuzzy for many researchers and practitioners, who often confuse it with the Hadoop technology. Thus, we provide in this paper a comprehensive state of the art of the different approaches to data lake design. We particularly focus on data lake architectures and metadata management, which are key issues in successful data lakes. We also discuss the pros and cons of data lakes and their design alternatives.
1 Introduction
The paper addresses big data challenges that exceed traditional management systems and clarifies the ambiguous data lake concept through a broad survey focused on design, architectures, and metadata management.
- Exponential data growth from IoT and social media produces volume, speed, heterogeneous-source, and structural challenges for traditional systems.
- A data lake stores, manages, and analyzes very large datasets in any format, but its concept remains confused with Apache Hadoop.
- Existing reviews are brief, topic-specific, or purely industrial, motivating a more comprehensive state of the art of data lake design.
- The survey reviews definitions, architectures, implementation technologies, metadata management, and the advantages and disadvantages of data lakes.
- The paper excludes data ingestion, governance, and security because these topics are currently little addressed in the literature.
2 Data Lake Definitions
The paper surveys competing data lake definitions and proposes a scalable definition that combines native-format data, metadata, governance, broad access, organization, and scalability.
- Data lakes were initially described as repositories for raw, heterogeneous data from multiple sources that support exploration, extraction, and analysis.
- Some literature equated data lakes with Hadoop, whereas later work also associates them with proprietary clouds, NoSQL systems, multistores, and data-driven design patterns.
- The consensual variety and schema-on-read definition stores all formats without a strict schema, leaving schema requirements until querying.
- The authors consider that definition fuzzy, then amend a more precise proposal by adding scalability for big-data storage and processing.
- Their definition describes a scalable system retaining data of any type in native format, mainly for data specialists seeking knowledge extraction.
- Required characteristics include metadata-based quality enforcement, governance policies, varied user access, all data types, logical and physical organization, and scalable storage and processing.
3 Data Lake Architectures and Technologies
The paper argues that existing pond-versus-zone classifications are fuzzy, proposes a Functional × Maturity typology, reviews implementation technologies, and examines data-lake/data-warehouse association.
- Existing architecture reviews commonly distinguish pond and zone approaches, but the paper finds this categorization sometimes fuzzy.
- The authors introduce a Functional × Maturity classification to provide a new way to categorize data lake architectures.
- The architecture discussion also reviews possible implementation technologies and investigates how data lakes can be associated with data warehouses in enterprise architectures.
3.1 Data Lake Architectures
The paper reviews pond and zone architectures, then proposes classifying data lakes by functionality and data maturity. It argues that hybrid architectures are preferable because single-view approaches overlook either required functions or the data lifecycle, while existing hybrids retain limitations.
- Pond architectures: Pond architectures divide lakes into specialized data ponds, each associated with storage, processing, conditioning, and analysis services.The raw data pond is a transit zone, while other ponds handle analog, application, textual, or archival data.
- Zone architectures: Zone architectures assign data to zones according to refinement level, but variants differ in zone number and characteristics.Zaloni’s architecture includes transient loading, raw, trusted, discovery sandbox, and consumption areas.
- Zone architectures: Lambda architecture separates batch processing for bulk data from real-time processing for fast IoT data.These specialized processing zones handle fast and bulk data in adapted ways.
- Architecture trade-offs: Both pond and zone architectures pre-process data for quick, easy analysis, but pond architectures can lose data when raw data are deleted during transfer.Zone variants can also create multiple copies that complicate lineage, while lambda architectures require cross-analysis across distinct processing logics.
- Functional × Maturity Architectures: The proposed typology distinguishes functional, data maturity-based, and hybrid architectures to overcome contradictions in the pond/zone categorization.Functional architectures organize components by lake functions, whereas maturity-based architectures organize them by data refinement.
- Functional × Maturity Architectures: Functional architectures align components with technologies, while maturity-based architectures organize the data lifecycle; both are limited by their single viewpoint.Hybrid architectures combine data functions and refinement, but existing hybrids may delete raw data or restrict access to refined data.
3.2 Technologies for Data Lakes
The paper reviews technologies supporting data-lake ingestion, storage, processing, querying, and user access beyond the commonly assumed Hadoop ecosystem. It presents complementary tools because data lakes handle heterogeneous data and workloads that no single technology addresses completely.
- Technology landscape: Hadoop implementations combine HDFS storage with MapReduce or Spark processing, but Hadoop is not the only suitable data-lake technology.The review therefore considers tools for each basic data-lake function beyond Hadoop.
- Data ingestion: Ingestion tools transfer source data into lakes and may also aggregate, convert, clean, or retrieve data through common protocols and APIs.Examples include Flink, Samza, Flume, Kafka, wget, rsync, FTP, HTTP, CKAN, and Socrata.
- Data storage: Relational DBMSs store structured data, whereas NoSQL and multi-paradigm approaches better accommodate semi-structured and unstructured variety.Relational systems are described as ill-adapted to semi-structured and especially unstructured data.
- Data storage: 75% of data lakes use HDFS storage, which provides scalable, fault-tolerant, schema-free bulk storage for all data types.HDFS alone is insufficient for some structured formats and should ideally be combined with relational and/or NoSQL DBMSs.
- Data processing: MapReduce suits very large data but is less efficient for fast data, while Spark, Flink, and Storm support real-time processing.MapReduce uses disk-based intermediate results; Spark uses an in-memory approach and both approaches can coexist for different workloads.
- Data access: Classical query languages cannot query across heterogeneous databases simultaneously, motivating multistore and query-rewriting technologies.Examples include Spark SQL, SQL++, SQRE, CloudMdsQL, and Apache Phoenix.
3.3 Combining Data Lakes and Data Warehouses
Data lakes and data warehouses can be combined either by sourcing the warehouse from the lake or by embedding warehouse components within a lake. These alternatives balance flexible storage and analysis with integration and siloing concerns.
- Two main integration approaches are identified: data lakes can source data warehouses, or data warehouses can function as data-lake components.
- Data lake sourcing a data warehouse: Using a data lake as a staging area combines cheaper large-scale raw-data storage with warehouse-based industrialized OLAP and lake-based ad-hoc analysis.
- Data warehouses as data-lake components: Inmon’s approach subdivides data lakes into data ponds, including structured ponds that function as data warehouses.
- Discussion: The sourcing approach separates industrialized warehouse analyses from on-demand lake analyses but introduces a data-siloing issue.
- Discussion: Inmon’s unified platform can reduce siloing, support cross-reference analyses, and improve tracking and reproducibility of warehouse data processes.
4 Metadata Management in Data Lakes
Metadata management is essential because schema-free lake data can otherwise produce data swamps. The paper reviews functional and structural classifications and favors structural categorization as broader, while noting ambiguity in functional categories.
- Without an efficient metadata system, schema-free data ingested into lakes can turn them into data swamps.
- The literature distinguishes functional metadata from classifications based on structural metadata types.
- Functional metadata: Oram’s classification groups metadata by collection process into business, operational, and technical metadata.
- Structural metadata: Sawadogo et al.’s object-based classification covers intra-object, inter-object, and global metadata.
- Discussion: The paper favors structural categorization because it extends and generalizes functional metadata, while Oram’s categories can overlap and confuse practitioners.
4.2 Metadata Modeling
Data-lake metadata is represented mainly through graph models or data-vault models. Graphs support provenance, similarity, and composition, while data vaults emphasize schema evolution and relational implementation.
- The literature identifies graph-based and data-vault approaches for representing data-lake metadata.
- Graph-based models: Provenance-centered graphs track activities, data objects, and users, typically using directed acyclic graphs.
- Graph-based models: Similarity-centered graphs connect data objects with weighted or unweighted edges to represent likeness and support network analyses.
- Graph-based models: Composition-centered graphs decompose objects into elements such as columns and tags, helping users navigate data and detect connections.
- Data-vault models: Data vaults support schema evolution through hubs, links, and satellites, and can be implemented in relational DBMSs, although adaptations are needed for graph-like data linkage.
- Discussion: Graph models can enrich future analyses, but integrating provenance, similarity, and composition approaches remains an open issue.
4.3 Metadata Generation
Metadata generation combines ingestion tools, specialized extraction technologies, ad-hoc algorithms, and metadata frameworks. These methods extract properties, provenance, schemas, links, entities, and lineage from lake data.
- Data-ingestion tools can also extract metadata, including provenance through Apache Flume and properties or semantic metadata through CKAN protocols.
- Apache Tika detects object MIME types and languages, while Open Calais and IBM’s Alchemy API identify entities, infer relationships, and detect events.
- Ad-hoc algorithms can detect links between attributes and discover schemas or constraints in semi-structured data.
- Apache Atlas uses hooks based on logs to extract lineage metadata and propagate tags across derivations of tagged data.
4.4 Features of Data Lake Metadata Systems
The survey evaluates metadata systems through six desired features and compares eighteen state-of-the-art systems and models. MEDAL covers all six features, while Ground offers a detailed, nearly complete alternative.
- Feature framework: A comprehensive data lake metadata system should implement six identified features.The survey introduces these features as criteria for assessing metadata-system completeness.
- Feature framework: Semantic Enrichment adds titles, tags, descriptions, and related information so data become comprehensible and linkable.Knowledge bases such as ontologies are commonly used for semantic annotation.
- Feature framework: Data Versioning retains previous data states while managing updates, supporting reproducibility, inconsistency correction, branching, and concurrent evolution.Usage Tracking records user interactions such as creation, reading, and updating, but the passage provides only a partial description.
- System comparison: The comparison distinguishes theoretical metadata models from operational implementations, which are often less detailed about their methodology.The survey compares eighteen systems and models according to the proposed features.
- System comparison: MEDAL covers all six features, whereas GOODS and CoreKG implement five and Ground covers four while providing substantially more conceptual detail.GOODS and CoreKG are described as black-box systems; Ground is presented as much more detailed and almost as complete.
- System comparison: Data polymorphism and data versioning are concentrated in the most complete systems, likely because implementation complexity limits their broader adoption.The survey identifies these two features as advanced.
5 Pros and Cons of Data Lakes
Data lakes offer inexpensive, flexible, scalable storage and analysis for heterogeneous data, but their benefits depend on expertise, metadata, access services, and suitable technologies. The section contrasts these advantages with risks including inconsistency, ambiguity, immaturity, and difficult distributed analytics.
- Benefits and pitfalls: Data lakes cost ten to one hundred times less to deploy than traditional decision-oriented databases.The passage attributes this to open-source technologies such as HDFS and pay-for-use cloud storage.
- Benefits and pitfalls: The data-lake concept remains ambiguous because it is often treated as synonymous with, or a marketing label for, HDFS.This misconception persists despite the broader range of technologies associated with data lakes.
- Benefits and pitfalls: Data fidelity preserves original data and avoids preprocessing losses, but integrating disparate sources without transformation increases inconsistency risk.The trade-off follows directly from retaining raw data while combining multiple sources.
- Benefits and pitfalls: Schema-on-read and support for unstructured data broaden analytical flexibility, while immature standards and missing access services can impede knowledge extraction.Data lakes accommodate varied formats, but explicit schemas are absent and access services or APIs are essential.
- Benefits and pitfalls: Real-time ingestion avoids source-to-lake transformation delays, but efficient metadata management is required to keep the lake usable.Inappropriate metadata methods or technologies can turn the lake into an inoperable data swamp.
- Benefits and pitfalls: Distributed data-lake technologies provide scalability and replication-based resilience against hardware and software failures.Examples include HDFS, MapReduce, Apache Spark, and Elasticsearch.
- Benefits and pitfalls: Data lakes require specialized, more expensive data-science expertise for on-the-fly analysis, unlike traditional systems operated by business users.The section characterizes lakes as analyst sandboxes and contrasts their typical users with traditional decision systems.
- Benefits and pitfalls: Heterogeneous data integration enables discovery of links and correlations, but classical statistical and AI methods may require difficult adjustment for distributed or streaming environments.The passage specifically cites K-means and K-Nearest Neighbors as examples needing readjustment.
6 Conclusion
The conclusion presents the survey as a comprehensive state of the art on data-lake design, architectures, metadata management, and trade-offs. It identifies unresolved challenges in integration, querying, unstructured-data analysis, governance, and privacy.
- Contributions: The survey synthesizes approaches for designing and conceptually building data lakes.Its coverage includes definitions, architectures, technologies, metadata techniques, benefits, drawbacks, and a summary mind map.
- Scope and open issues: The paper opens current research issues rather than addressing data ingestion, governance, and security as developed survey topics.These topics were excluded because the literature currently addresses them only sparsely.
- Scope and open issues: Data integration and transformation remain difficult for big data, while User-Defined Functions are harder to optimize than classical queries.The conclusion states that this optimization issue is not yet addressed by the literature.
- Scope and open issues: Metadata-based interrogation remains challenging because multistores and polystores do not cover unstructured data, and virtual integration may require new big-data-tailored approaches.Unstructured data are commonly queried separately through index stores.
- Scope and open issues: Unstructured data beyond text, including images, sounds, and videos, lack envisioned global querying and analysis solutions in the literature.The conclusion describes these modalities as insufficiently addressed in data-lake research.
- Scope and open issues: Data governance principles are seldom translated into solutions for data quality, security, lifecycle management, and metadata lineage.These concerns are often treated as risks rather than addressed proactively.
- Scope and open issues: Data lakes' large-scale cross-analysis can create serious privacy breaches, leaving privacy issues under active research.The conclusion distinguishes these risks from technical security controls and GDPR-related concerns.