Source-linked AI summary
DataHub: Collaborative Data Science & Dataset Version Management at Scale
Anant Bhardwaj, Souvik Bhattacherjee, Amit Chavan, Amol Deshpande, Aaron J. Elmore, Samuel Madden, Aditya G. Parameswaran
TL;DR
Collaborative teams need better support for managing large, heterogeneous, evolving datasets than relational databases, file systems, and existing version-control systems provide. The paper proposes DSVC for multi-version dataset management and DATAHUB as a hosted collaborative analysis platform built on it, while outlining the research challenges of scaling these capabilities.
Problem
Existing systems inadequately support collaborative work on large, heterogeneous, evolving datasets, leading teams toward ad hoc sharing and manual version management.
Method
The paper proposes DSVC for dataset version management and DATAHUB, a hosted platform built on DSVC for collaborative data analysis.
Results
The paper introduces the integrated systems, outlines challenges in managing and querying large multi-version datasets, and proposes initial solutions.
Takeaways & Limitations
Addressing these dataset-versioning challenges is presented as essential for supporting large-scale collaborative data analytics.
Takeaways & Limitations
Efficient dataset version control remains challenging because arbitrary versions may differ across tables and schemas, while version graphs require nontrivial storage choices.
Abstract
from arXiv · showhide
Relational databases have limited support for data collaboration, where teams collaboratively curate and analyze large datasets. Inspired by software version control systems like git, we propose (a) a dataset version control system, giving users the ability to create, branch, merge, difference and search large, divergent collections of datasets, and (b) a platform, DataHub, that gives users the ability to perform collaborative data analysis building on this version control system. We outline the challenges in providing dataset version control at scale.
1. INTRODUCTION
The paper frames collaborative data analysis as poorly supported by existing systems and proposes DSVC plus DATAHUB to manage, query, and share evolving datasets at scale.
- Motivation: Collaborative data teams work with diverse, evolving datasets through ad hoc sharing, trial-and-error analysis, and manual version management.Datasets range from structured to unstructured and complete to noisy or incomplete, while collaboration involves repeated back-and-forth among researchers.
- Motivation: Existing relational databases and source-code version control systems do not adequately support large, heterogeneous datasets or collaborative version management.Teams often resort to duplicated files and manually named versions distributed across storage systems.
- Contribution: DSVC is proposed as a common substrate for capturing modifications, reducing storage costs, querying versions, identifying differences, and sharing datasets.Its intended scope includes structured and unstructured datasets and richer version-management functionality than conventional source-control tools.
- Contribution: DATAHUB builds on DSVC as a hosted platform with tools for data cleaning, search and integration, visualization, and collaborative analysis.The paper compares DSVC and DATAHUB conceptually with git and GitHub while emphasizing broader dataset support and richer querying.
- Motivation: A survey of computational-biology groups found teams of 20-30 researchers sharing approximately 100 TB, costing nearly $100K per year in local storage.Researchers also faced unknown duplication, limited visibility into data use, and pressure to delete intermediate versions.
- Prior Work and Gap: Prior dataset-versioning work mainly handled linear version chains, whereas collaborative workflows require branching, merging, and comparing divergent versions.The paper identifies these capabilities as part of the novel research challenges addressed by DSVC.
2. PRIOR WORK
Prior systems provide source-code or specialized dataset versioning, but they do not adequately scale to large datasets or offer the richer version queries targeted by DSVC.
- Source-Code Version Control: Git and SVN are designed for modest-sized files and perform poorly on collections of large files.In the reported test, git ran out of memory while SVN took about 20x longer than loading and copying an uncompressed raw file.
- Data-Science Systems: Most data-science, workflow, and collaboration tools lack dataset version-management capabilities.The paper positions DSVC as a possible backend data-management layer for these systems.
- Data-Management Projects: Emerging dataset-management projects show practitioner demand but, according to the paper, do not provide its proposed rich versioning and querying functionality.The cited systems include CKAN, Domo, Enterprise Data Hub, Domino, Amazon Zocalo, and Dat.
- Specialized Versioning: Research on versioning graphs and arrays offers preliminary techniques, while broader database dataset versioning remains relatively limited.The paper proposes extending this line toward general structured datasets and compact difference encoding and querying.
3. PROPOSED DESIGN AND INTERFACES
DSVC provides dataset version management and querying, while DataHub builds a hosted collaborative analysis platform on top of it. The design supports flexible data representations, version graphs, branching, merging, local or transactional access, and sampling for large datasets.
- System architecture: The architecture centers on DSVCP for version processing and VQP for querying multiple dataset versions through VQL.VQL is described as an enhanced SQL interface for multi-version queries.
- System architecture: DSVC is a multi-version dataset management substrate, and DataHub is a hosted platform adding richer interaction and data-cleaning tools.DataHub exposes DSVC’s versioning API and VQL to applications.
- DSVC data model: DSVC uses schema-later records and datasets composed of related tables, allowing structured and unstructured data to coexist.Records have keys and typed, named attributes, while datasets include table-level connection structure such as foreign keys.
- DSVC data model: Dataset versions are represented as a directed acyclic version graph whose edges capture version, branching, derivation, and provenance relationships.Users or applications may supply provenance metadata when creating versions.
- Versioning interfaces: The versioning API supports creating, branching, merging, committing, and rolling back datasets, with workflows such as staged cleaning and integration.The API is intended for both end users and DataHub components.
- Access and collaboration: Users can work transactionally or through local copies, with lazy fetching for large datasets and sampled checkouts whose changes are applied during merging.Sample merges can report constraint violations or merge conflicts.
- Access and collaboration: A central research challenge is efficiently detecting and resolving conflicts across divergent branches, including disjoint attribute updates on overlapping rows.The design aims to merge non-conflicting changes while guiding users through conflicting merges.
- Query language: Making VQL complete and easy to use remains an open design challenge, including operations over retrieved version sets and version subgraphs.The supplied query example is described as somewhat unwieldy.
4. STORAGE REPRESENTATIONS
The paper presents version-first and record-first representations for branching dataset histories, each favoring different retrieval and query workloads. Efficient dataset version management requires balancing storage compactness, retrieval cost, query performance, and archival constraints.
- Version-First Representation: Version-first representation stores each version’s records, potentially as deltas from parent versions, making complete-version checkout straightforward.A storage graph can combine materialized versions with deltas and support retrieval by walking the graph.
- Version-First Representation: Recording deltas is difficult when versions use different tables or schemas, especially when external transformations are unknown.Binary differencing can be memory-intensive and slow on large files, motivating database-aware record-level approaches.
- Version-First Representation: Version-graph encoding must choose delta relationships independently of graph adjacency because nonadjacent versions may have smaller differences.The system must quickly estimate pairwise difference sizes to choose storage relationships effectively.
- Version-First Representation: Materializing versions avoids expensive delta-chain traversal, while Steiner datasets and optimization techniques can reduce retrieval costs.The proposed optimization considers pairwise encodings and may add extra datasets as retrieval shortcuts.
- Record-First Representation: Record-first representation stores each record with a compressed list of versions containing it, supporting efficient version discovery through record-property indexes.It also supports shared computation for queries over groups of versions, while complete retrieval may be less efficient.
- Design Trade-offs: DSVC needs both representations because complete-version retrieval and property-based version identification favor different storage layouts.This reflects a broader efficiency-versus-storage tradeoff, alongside the need for archival and cleanup to meet storage budgets.
5. CONCLUSIONS
The paper introduces DSVC and DATAHUB and frames large-scale multi-version dataset management as necessary for collaborative data analytics. It proposes initial solutions while outlining challenges beyond ordinary source-code version control.
- Conclusions: The paper introduces a dataset version control system and DATAHUB as tightly integrated systems for collaborative data analytics.The design draws an analogy with git and GitHub while targeting large multi-version datasets.
- Conclusions: Large multi-version datasets create management and querying challenges that do not arise in regular source-code version control.The paper presents initial solutions and argues that addressing these challenges is essential for large-scale collaborative data analytics.