Source-linked AI summary
AsterixDB: A Scalable, Open Source BDMS
Sattam Alsubaiee, Yasser Altowim, Hotham Altwaijry, Alexander Behm, Vinayak Borkar, Yingyi Bu, Michael Carey, Inci Cetindil, Madhusudan Cheelangi, Khurram Faraaz, Eugenia Gabrielova, Raman Grover, Zachary Heilbron, Young-Seok Kim, Chen Li, Guangqiang Li, Ji Mahn Ok, Nicola Onose, Pouria Pirzadeh, Vassilis Tsotras, Rares Vernica, Jian Wen, Till Westmann
TL;DR
AsterixDB addresses the need to manage and access large volumes of semi-structured data without assembling multiple narrow systems. The paper describes its integrated BDMS design and reports that early AsterixDB was competitive with more mature systems across several query types and batched insertions, while noting important early-system limitations.
Problem
Organizations need systems that can store, manage, and access the growing volume of semi-structured Big Data generated across many domains.
Method
The paper presents AsterixDB’s flexible data model, query language, architecture, ingestion and storage facilities, indexing, and transactional support as one integrated BDMS.
Results
Early AsterixDB was surprisingly competitive with more mature systems for various query types and batched insertions.
Takeaways & Limitations
AsterixDB’s preliminary results suggest that broad functionality can coexist with basic system performance across the evaluated workloads.
Takeaways & Limitations
AsterixDB lacks a cost-based query optimizer and instead relies on safe rewrite rules, partitioning awareness, and user-provided optimization hints.
Abstract
from arXiv · showhide
AsterixDB is a new, full-function BDMS (Big Data Management System) with a feature set that distinguishes it from other platforms in today's open source Big Data ecosystem. Its features make it well-suited to applications like web data warehousing, social data storage and analysis, and other use cases related to Big Data. AsterixDB has a flexible NoSQL style data model; a query language that supports a wide range of queries; a scalable runtime; partitioned, LSM-based data storage and indexing (including B+-tree, R-tree, and text indexes); support for external as well as natively stored data; a rich set of built-in types; support for fuzzy, spatial, and temporal types and queries; a built-in notion of data feeds for ingestion of data; and transaction support akin to that of a NoSQL store. Development of AsterixDB began in 2009 and led to a mid-2013 initial open source release. This paper is the first complete description of the resulting open source AsterixDB system. Covered herein are the system's data model, its query language, and its software architecture. Also included are a summary of the current status of the project and a first glimpse into how AsterixDB performs when compared to alternative technologies, including a parallel relational DBMS, a popular NoSQL store, and a popular Hadoop-based SQL data analytics platform, for things that both technologies can do. Also included is a brief description of some initial trials that the system has undergone and the lessons learned (and plans laid) based on those early "customer" engagements.
1. OVERVIEW
AsterixDB addresses the challenge of managing and accessing massive semi-structured data by combining broad data-management capabilities in a scalable BDMS. The paper introduces its design, architecture, and early evaluation.
- AsterixDB targets the storage, management, indexing, querying, and analysis of mass quantities of semi-structured data on shared-nothing commodity clusters.
- Its BDMS design aims to support flexible schemas, SQL-power queries, parallel execution, data management, automatic indexing, continuous ingestion, large-scale clusters, and common Big Data types.
- AsterixDB seeks to cover capabilities that existing Hadoop-based query platforms, NoSQL stores, and parallel relational DBMSs lack in combination.
- The system accepts data through loading, continuous feeds, or insertion queries, and provides synchronous or asynchronous query-result access through its cluster architecture.
- The paper presents AsterixDB’s data model, query language, software architecture, project status, performance comparisons, and lessons from early user trials.
2. DATA DEFINITION
AsterixDB’s data-definition layer organizes semi-structured data with flexible ADM types, dataverses, datasets, indexes, external data access, feeds, and reusable functions. These features support both structured and evolving data while preserving management and query integration.
- Dataverses, Datatypes, and Datasets: A dataverse contains application-specific types, datasets, functions, and other artifacts, while datasets store instances conforming to their associated datatype.
- Dataverses, Datatypes, and Datasets: Open datatypes permit additional instance fields beyond the declared schema, while closed datatypes restrict instances to listed fields and can mark fields optional.
- Dataverses, Datatypes, and Datasets: ADM extends JSON with richer types, constructs such as bags, and a schema language for modeling semi-structured data.
- Advanced Types and Functions: AsterixDB supports advanced spatial, temporal, and textual types and functions, including features designed for Big Data applications.
- Dataset and Index Creation: Datasets use primary-key B+-trees, secondary indexes keyed through primary keys, and primary-key hash partitioning across cluster nodes.
- External Data: External datasets can be queried like internal datasets without preloading data, but the current release makes them read-only and static and disallows indexes.
- Data Feeds and User Defined Functions: Built-in data feeds continuously ingest external data into datasets and their indexes, while user-defined functions provide reusable parameterized query views.
3. DATA MANIPULATION
AQL is a declarative expression language based loosely on XQuery, designed for nested, semi-structured ADM data and a broad range of queries. Its examples cover dataset access, filtering, joins, nesting, fuzzy matching, quantification, aggregation, and external log data.
- AQL design: AQL is loosely based on XQuery and preserves semantics with and without schemas while supporting nesting and semi-structured ADM data.Its design removes XML-specific features such as document order and node identity.
- Core query constructs: FLWOR expressions organize common queries through for, let, where, return, and order by clauses, with additional group by and limit support.For binds instances from sequences, while let binds variables to entire intermediate result expressions.
- Filtering and access: AQL supports filtering records by datetime ranges, including queries that may use an index on the filtered field during evaluation.The example selects users who joined between July 22, 2010 and July 29, 2012.
- Joins and nesting: Nested FLWOR expressions provide left outer-join semantics and return nested ADM records, including users with no matching messages.AQL can also synthesize flat records through equijoins between users and messages.
- Advanced predicates: AQL includes fuzzy matching with configurable edit-distance semantics, existential quantification over nested data, and access to open fields absent from declared types.The examples use edit distance less than or equal to 3 and test employment records with null end dates and an open job-kind field.
- Aggregation and fuzzy joins: AQL supports aggregation and fuzzy joins, including average message length, country-grouped active-user counts using external logs, and Jaccard similarity of 0.3.AsterixDB provides both ad hoc parallel fuzzy joins and indexed fuzzy joins.
- Data modification: Data modification currently replaces records through delete-then-insert operations, reflecting an initial focus on append-heavy rather than modification-heavy use cases.This scope boundary accompanies AsterixDB’s record-level transaction model.
- Transactions: AsterixDB supports implicit record-level ACID transactions but does not support multi-statement transactions.AQL statements involving multiple records can therefore comprise multiple independent record-level transactions.
4. SYSTEM ARCHITECTURE
AsterixDB’s architecture layers query compilation and optimization over the scalable Hyracks runtime, with storage, indexing, ingestion, transactions, and external-data support integrated for distributed execution.
- Query control and execution: The Query Control Node accepts HTTP queries, compiles AQL into Hyracks job descriptions, distributes them to worker nodes, and returns results synchronously or asynchronously.Asynchronous requests receive a handle for status checks and result retrieval.
- Hyracks runtime: Hyracks executes data-parallel DAGs of Operators and Connectors, with a library of 53 Operators and 6 Connectors for joins, aggregations, and related processing.Operators consume and produce partitions, while Connectors redistribute data between operators.
- Hyracks runtime: Query 10 combines local and global aggregation: local operators pre-aggregate in parallel, then a single global operator receives replicated results to reduce network traffic.Secondary-index results feed primary-index searches after primary-key sorting, while the aggregation split maximizes distributed computation.
- Query control and execution: AQL queries are translated into Algebricks programs, optimized through algebraic rewrites and partitioned parallelism, then generated as Hyracks jobs.The layered stack also supports Hadoop MapReduce jobs and the Pregelix programming model through Hyracks.
- Storage and indexing: AsterixDB uses LSM trees for all internal storage and indexes, buffering updates in memory and flushing and merging components to support high ingestion rates.The LSMification framework supports B+-trees, R-trees, and inverted keyword and n-gram indexes.
- Storage, ingestion, and transactions: Datasets use hash-partitioned primary-key B+-trees with node-local secondary indexes, while external datasets can be queried without loading data into AsterixDB.The compiler attempts to co-locate HDFS query tasks with their associated input data; record-level transactions span multiple LSM indexes with node-local 2PL.
5. STATUS AND PERFORMANCE
AsterixDB progressed through stabilization releases, planned major features, user pilots, and preliminary performance evaluations. Early results indicate competitive query and batched-insertion performance while exposing optimization and implementation limitations.
- Releases: Three public releases through 0.8.5 improved stability and performance; two larger 2014 releases planned feeds and indexes over external datasets.The 0.8.0 beta appeared in June 2013, followed by 0.8.3 and 0.8.5 at roughly five-month intervals.
- Limitations: AsterixDB lacks a cost-based optimizer, instead using approximately 90 rule-based rewritings and user-specified optimization hints.Its rules track partitioning and choose strategies such as index-based selections and parallel hash joins, while hints can override selected choices.
- User pilots: Pilot projects on cell-phone events and social-media analytics drove improvements in grouped aggregation, materialization, and AQL implementation.The pilots exercised windowed and spatial aggregation, sequential-event predicates, large data volumes, and interactive analysis.
- Experimental setup: AsterixDB’s evaluation used synthetic users, messages, and tweets, comparing Schema and KeyOnly open data types against System-X, MongoDB, and Hive.Queries and inserts were selected to cover operations supported by most evaluated systems; AsterixDB used its REST API.
- Insertion performance: With batch size 20, AsterixDB’s average insert time per record outperformed the other systems, despite worse single-record insertion performance.Single-record overhead mainly arose from Hyracks job generation and startup; bulk load and feeds can provide still better performance for huge volumes.
- Preliminary results: Early AsterixDB was competitive with more mature systems across query types and batched insertions, including no-index scans and indexed queries.The authors interpret these preliminary results as evidence that AsterixDB’s broader feature set need not sacrifice basic system performance.
6. SUMMARY AND FUTURE WORK
AsterixDB is presented as a full-function BDMS combining capabilities associated with analytics platforms, parallel RDBMSs, and NoSQL stores while remaining distinct from each. Its feature set includes scalable execution, indexed storage, specialized types and queries, continuous ingestion, and NoSQL-style ACIDity.
- AsterixDB combines analytics, relational, and NoSQL capabilities but differs from each through its integrated full-function BDMS design.It natively stores and indexes data, supports complex nested and flat data, and provides declarative queries across multiple datasets.
- AsterixDB provides scalable runtime execution, LSM-based storage, multiple index types, specialized data types, fuzzy operations, feeds, and NoSQL-style ACIDity.Its indexes include B+ trees, R trees, keyword indexes, and n-gram indexes; its specialized types include spatial, temporal, and textual types.
- Initial comparisons with Hive, a commercial parallel DBMS, and MongoDB found that AsterixDB performed surprisingly well for a new system.The authors planned further improvements based on this performance study and pilot engagements with early users.