PaperlayerView exact revision on arXiv

Source-linked AI summary

Apache Calcite: A Foundational Framework for Optimized Query Processing Over Heterogeneous Data Sources

Edmon Begoli, Jesús Camacho Rodríguez, Julian Hyde, Michael J. Mior, Daniel Lemire

arXiv:1802.10233v1cs.DB

TL;DR

Specialized data-processing systems need shared query optimization, language support, and efficient querying across heterogeneous sources. Calcite addresses these needs with an embeddable framework combining relational planning, adapters, extensible optimization, and support for multiple processing models. Its flexible design has made it widely adopted across open-source big-data frameworks, although performance evaluation remains difficult because fair comparisons and heterogeneous benchmarks are limited.

  • Problem

    Specialized systems independently develop query optimization and language support, while users need optimized queries across heterogeneous data sources.

  • Method

    Calcite combines relational query processing, adapters for external sources, and a pluggable optimizer that reasons about physical properties and execution conventions.

  • Results

    Calcite’s flexible optimizer and adapter architecture support heterogeneous data processing and have made it the most widely adopted query optimizer in many open-source frameworks.

  • Takeaways & Limitations

    Calcite can serve selectively as a common query-processing and optimization layer across conventional, semi-structured, streaming, and geospatial data systems.

  • Takeaways & Limitations

    Calcite’s performance is difficult to evaluate fairly because systems may use different execution engines and heterogeneous benchmarks are limited.

Abstract

from arXiv · show

Apache Calcite is a foundational software framework that provides query processing, optimization, and query language support to many popular open-source data processing systems such as Apache Hive, Apache Storm, Apache Flink, Druid, and MapD. Calcite's architecture consists of a modular and extensible query optimizer with hundreds of built-in optimization rules, a query processor capable of processing a variety of query languages, an adapter architecture designed for extensibility, and support for heterogeneous data models and stores (relational, semi-structured, streaming, and geospatial). This flexible, embeddable, and extensible architecture is what makes Calcite an attractive choice for adoption in big-data frameworks. It is an active project that continues to introduce support for the new types of data sources, query languages, and approaches to query processing and optimization.

1 INTRODUCTION

Apache Calcite addresses duplicated query-processing needs in specialized systems and the difficulty of optimizing queries across heterogeneous data sources. Its adoption is supported by an extensible architecture, multiple data models, and a flexible optimizer.

  • Motivation: Specialized systems repeatedly need query optimization and support for SQL, streaming queries, and language-integrated queries.Calcite was developed to provide common query execution, optimization, and query-language functionality while leaving storage and data management to specialized engines.
  • Motivation: Organizations often need optimized queries spanning systems such as Elasticsearch, Apache Spark, and Druid.Calcite exposes a common interface for cross-platform optimization and supports global decisions such as materialized-view selection.
  • Design challenge: Calcite must remain extensible and flexible enough to integrate different types of systems.The framework’s common-layer role creates architectural challenges because integrated systems can differ substantially.
  • Adoption factors: Calcite’s open-source, Apache-backed Java implementation supports collaborative development and interoperability with JVM-based data-processing systems.The paper identifies open-source friendliness as a feature contributing to Calcite’s adoption.
  • Adoption factors: Calcite supports conventional and streaming processing, treating streams as time-ordered sets of records or events rather than persisted disk data.Its multiple-data-model support covers both streaming and conventional processing paradigms.
  • Adoption factors: Calcite’s optimizer is pluggable and extensible, including configurable rules, cost models, and multiple planning engines.Optimization can be divided into phases handled by different optimization engines.

2 RELATED WORK

Related systems provide modular optimization, federated querying, or broad data-model support, but Calcite combines these roles through a unifying relational abstraction and cross-backend optimization.

  • Modular optimizers: Orca modularizes optimization and exchanges information with execution engines, whereas Calcite can federate multiple storage and processing backends.Calcite also supports pluggable planners and optimizers and can operate as a standalone query execution engine.
  • Related query systems: Spark SQL queries multiple data sources, but its Catalyst optimizer lacks Calcite’s dynamic programming approach and risks local minima.The comparison concerns optimizer strategy rather than SQL support.
  • Modular optimizers: Algebricks provides a data-model-agnostic algebraic compiler layer, while Calcite additionally supports cost-based optimization.Both systems use modular approaches, but their optimization capabilities differ as described here.
  • Federated systems: Garlic unifies data from multiple systems but does not optimize queries across them, relying on each system’s own optimizer.This distinguishes Garlic’s unified object model from Calcite’s cross-system optimization role.
  • Federated systems: FORWARD uses SQL++ over a semi-structured model combining JSON and relational data, whereas Calcite represents semi-structured data relationally during planning.Both systems decompose or represent federated queries across underlying databases, but their data-model abstractions differ.
  • Federated systems: BigDAWG restricts cross-storage querying to an island with its own language and data model, while Calcite provides a unifying relational abstraction across backends.The distinction is the boundary of the abstraction used for cross-system queries.

3 ARCHITECTURE

Calcite supplies database-processing components without storage, execution algorithms, or metadata repositories, allowing it to mediate among applications, storage locations, and processing engines. Its architecture parses queries into relational operators, optimizes them, and maps the result back to an execution system.

  • Architecture: Calcite deliberately omits data storage, processing algorithms, and metadata repositories while retaining many typical database-management components.These omissions make it suitable for mediating applications that use multiple storage locations and processing engines.
  • Architecture: The optimizer represents queries as trees of relational operators and primarily uses rules, metadata providers, and planner engines.Figure 1 presents these components and their possible external interactions.
  • Query processing: A parser and validator translate SQL into relational-operator trees, while adapters define table schemas and views in external storage engines.Calcite can therefore operate on top of storage engines without containing a storage layer itself.
  • Query processing: Calcite can optimize SQL for systems with limited optimization and translate the optimized relational expression back into SQL.This allows it to operate as a standalone layer over systems that expose SQL interfaces but lack an optimizer.
  • Query processing: After optimization, applications retrieve the optimized relational expression and map it back to their query-processing unit.This interface supports integration with systems that use their own language parsing and interpretation.

4 QUERY ALGEBRA

Calcite represents queries with relational algebra and optimizes alternative plans using operators, physical traits, calling conventions, and planner rules. This design supports execution across heterogeneous engines, including optimized cross-system joins.

  • Relational algebra: Relational algebra forms Calcite’s core representation, with operators for common data manipulation and complex analytic operations such as windows.The window operator encapsulates bounds, partitioning, and aggregate functions for each window.
  • Traits: Physical traits describe properties such as ordering, grouping, and partitioning without changing the logical rows produced by an operator.Traits let the optimizer compare alternative plans and can enable removal of unnecessary operations, such as sorting already ordered input.
  • Traits: Calcite can enforce traits and convert expressions between trait values through converter interfaces implemented by relational operators.This supports planning around required physical properties such as column sort order.
  • Calling conventions: The calling convention trait identifies the backend where an expression executes, allowing transparent optimization of queries spanning different engines.A MySQL-to-Splunk join can be planned to run inside Splunk when its lookup capability is more efficient than using Spark as an external engine.

5 ADAPTERS

Calcite adapters provide a modular interface for incorporating diverse data sources while exposing backend-specific access paths and optimization rules. They support pushdown and execution across multiple backends.

  • Adapter architecture: An adapter incorporates a data source through a model, schema, schema factory, and tables that provide physical data access.The model specifies physical properties, the schema defines data formats and layouts, and the schema factory generates metadata-based schemas.
  • Adapter architecture: Adapters can add planner rules that convert logical relational expressions into expressions using the adapter’s calling convention.These rules connect Calcite’s logical algebra with backend-specific execution capabilities.
  • Execution interfaces: A table scan is the minimal adapter interface, while the enumerable convention supplies tuple-iterator operators for functionality unavailable in a backend.This lets Calcite implement operators such as joins outside individual adapter backends.
  • Optimization and pushdown: Adapter-specific rules can push filtering and other logic into a backend, assigning lower costs to backend-capable expressions.The same rule-based optimizer can therefore avoid enumerating all tuples when a query touches only a small subset of a table.
  • Optimization and pushdown: Adapters enable optimization across multiple backends by pushing possible logic into each backend before joining or aggregating the resulting data.Any relational-algebra expression can be pushed down through optimizer rules, from simple scans to advanced optimizations.

6 QUERY PROCESSING AND OPTIMIZATION

Calcite combines rule- and cost-based relational optimization with extensible metadata, planner engines, materialized-view rewriting, and adapter-specific pushdown. Its optimizer preserves query semantics while seeking lower-cost plans and supports both built-in and user-defined optimization components.

  • Optimizer architecture: Calcite repeatedly applies semantics-preserving planner rules to relational expressions, guided by a cost model that seeks lower-cost alternatives.The optimizer is the framework’s main component.
  • Extensibility: Every optimizer component is extensible, allowing users to add relational operators, rules, cost models, and statistics.
  • Planner rules: Calcite includes several hundred optimization rules, while dependent systems commonly add rules for backend-specific rewritings.
  • Planner rules: Cassandra sort pushdown requires a single-partition filter and a compatible sort prefix before converting logical operators to Cassandra-specific operators.The rule first requires the filter to be pushed down as a CassandraFilter.
  • Metadata providers: Metadata providers supply cost, cardinality, result size, parallelism, plan structure, and backend statistics used during optimization.Providers can override existing functions or add new metadata functions.
  • Planner engines: Calcite offers cost-based and exhaustive planner engines, and users can combine rule sets into consecutive optimization phases.The cost-based engine tracks alternative plans using a dynamic-programming algorithm; the exhaustive engine fires rules until no expression changes.
  • Materialized views: Materialized-view rewriting uses view substitution or lattice-based tiles to answer incoming queries from precomputed data.Lattice rewriting is efficient for star-schema expressions but more restrictive than view substitution.

7 EXTENDING CALCITE

Calcite extends relational query processing beyond ordinary SQL to semi-structured, streaming, geospatial, and Java language-integrated queries. These extensions represent diverse data abstractions through Calcite’s query language and relational planning infrastructure.

  • Semi-structured data: Calcite supports semi-structured data through nested ARRAY, MAP, and MULTISET column types with extraction using the [] operator.
  • Semi-structured data: MongoDB collections can be exposed as tables with a single _MAP column, enabling views that combine semi-structured and relational data.Document fields such as city, longitude, and latitude can be extracted from _MAP.
  • Streaming: Calcite provides first-class streaming SQL through STREAM, windowing, and stream-to-stream join extensions.The STREAM directive requests incoming records rather than records already received.
  • Streaming: Streaming aggregates and joins use analytic and named window functions, including sliding, tumbling, hopping, and session windows.Windowing makes blocking operators usable over unbounded streams.
  • Streaming: Streaming window aggregates require monotonic or quasi-monotonic expressions in GROUP BY or, for sliding and cascading windows, ORDER BY.
  • Geospatial data: Calcite’s geospatial extension adds a GEOMETRY type for points, curves, and polygons and targets OpenGIS SQL-interface compliance.The passage describes geospatial support as preliminary.
  • Language-integrated queries: LINQ4J provides a Java language-integrated query language following Microsoft LINQ conventions, allowing programmers to write query code in one language.

8 INDUSTRY AND ACADEMIA ADOPTION

Calcite is adopted by open-source and commercial systems either as an embedded library or through adapters that federate query processing. Its integrations span relational, semi-structured, streaming, search, and heterogeneous data-management settings.

  • Integration models: Projects integrate Calcite either by embedding it as a library or by implementing adapters that federate query processing.
  • Embedded Calcite: Embedded systems may use Calcite’s query language interface, Avatica JDBC driver, parser and validator, relational algebra, and enumerable operators.Table 1 catalogs these component choices and the execution engine used.
  • Adopted systems: Drill uses Calcite with a schema-free JSON document model and a SQL dialect extended for semi-structured data.
  • Adopted systems: Hive adopts Calcite as its rule- and cost-based optimizer, translating queries into Calcite operators before producing Hive physical algebra.Hive retains its own JDBC driver, parser, and validator.
  • Adopted systems: Kylin exposes materialized-view cubes in HBase so Calcite can rewrite queries to use them, with execution combining Calcite native operators and HBase.
  • Adopted systems: Streaming projects including Apex, Flink, Samza, and Storm use Calcite components to provide streaming SQL interfaces.
  • Adapters: Calcite adapters translate relational algebra into backend-specific interfaces including SQL dialects, CQL, Pig Latin, Java RDD calls, and REST requests.

9 FUTURE WORK

Calcite’s future work targets standalone-engine features, a more modular and user-programmable planner, new parametric approaches, and expanded adapter support. The paper also identifies missing execution evaluation and difficulties in fair, heterogeneous benchmarking.

  • Planned extensions: Standalone-engine support would add data definition languages, materialized views, indexes, and constraints.
  • Planned extensions: Planner work includes greater modularity and user-supplied planner programs that organize rules into planning phases.
  • Performance evaluation: Calcite’s performance testing module does not evaluate query execution, leaving system performance assessment unresolved.
  • Performance evaluation: Fair timing comparisons with systems such as Algebricks are difficult because the systems use different execution engines and architectural baselines.
  • Performance evaluation: Performance assessment requires separate benchmarks for Calcite within one system and across multiple heterogeneous systems.The former depends on system characteristics, while the latter is constrained by available heterogeneous benchmarks.

10 CONCLUSION

Calcite addresses increasingly heterogeneous data scenarios while preserving SQL-based relational processing. Its flexible optimizer and adapter architecture support broad adoption across open-source data frameworks.

  • Calcite supports relational, semi-structured, streaming, and geospatial data sources alongside SQL-based processing.This positions the framework across both traditional relational systems and evolving heterogeneous data environments.
  • Its design emphasizes flexibility, adaptivity, and extensibility in query optimization and system integration.The optimizer and adapter architecture allow frameworks to embed Calcite selectively.
  • Calcite has become widely adopted across open-source frameworks including Hive, Drill, MapD, and Flink.The passage attributes this adoption to Calcite’s dynamic optimizer and adapter architecture.
  • Calcite’s heterogeneous processing and extended relational functions are expected to continue improving in functionality and performance.
Loading 1802.10233v1…