PaperlayerView exact revision on arXiv

Source-linked AI summary

Time Series Management Systems: A Survey

Søren Kejser Jensen, Torben Bach Pedersen, Christian Thomsen

arXiv:1710.01077v1cs.DB

TL;DR

The paper addresses the need for specialized systems to store and analyze rapidly growing time-series data that general-purpose DBMSs cannot adequately handle. It systematically classifies and reviews TSMS research by architecture, functionality, use case, and processing capabilities, then synthesizes research directions and proposes a next-generation vision. The survey finds that distributed TSMSs generally rely on existing distributed storage systems, while internal-store systems are mainly centralized or proof-of-concept implementations.

  • Problem

    Growing sensor-generated time-series volume and velocity exceed the suitability of general-purpose DBMSs for storage and analysis.

  • Method

    The paper performs a systematic literature survey that classifies TSMSs by architecture and reviews their use cases, storage and query functionality, components, Stream Processing, and AQP.

  • Results

    Distributed TSMSs generally use existing distributed file systems or distributed DBMSs, whereas internal-store TSMSs are predominantly centralized and often serve embedded-device or proof-of-concept purposes.

  • Takeaways & Limitations

    The survey’s next-generation vision is a distributed TSMS with data-warehouse-like analytics, real-time updates, stream processing, interactive historical and incoming-data queries, and AQP.

  • Takeaways & Limitations

    Some surveyed systems have practical scope limitations, including in-memory systems without long-term disk storage and replicated systems without consistency guarantees.

Abstract

from arXiv · show

The collection of time series data increases as more monitoring and automation are being deployed. These deployments range in scale from an Internet of things (IoT) device located in a household to enormous distributed Cyber-Physical Systems (CPSs) producing large volumes of data at high velocity. To store and analyze these vast amounts of data, specialized Time Series Management Systems (TSMSs) have been developed to overcome the limitations of general purpose Database Management Systems (DBMSs) for times series management. In this paper, we present a thorough analysis and classification of TSMSs developed through academic or industrial research and documented through publications. Our classification is organized into categories based on the architectures observed during our analysis. In addition, we provide an overview of each system with a focus on the motivational use case that drove the development of the system, the functionality for storage and querying of time series a system implements, the components the system is composed of, and the capabilities of each system with regard to Stream Processing and Approximate Query Processing (AQP). Last, we provide a summary of research directions proposed by other researchers in the field and present our vision for a next generation TSMS.

1 INTRODUCTION

The survey motivates specialized TSMSs because large-scale sensor networks produce time series that general-purpose DBMSs handle poorly. It reviews prior systems and defines a structured search and analysis approach for surveying current TSMS research.

  • Motivation: Large sensor deployments produce time series at scales requiring efficient transfer, storage, and analysis.The paper notes examples ranging from household IoT devices to industrial systems and reports that a Boeing 787 can produce more than half a terabyte per flight.
  • Motivation: General-purpose DBMSs, especially RDBMSs, are unsuitable for the velocity and volume of contemporary sensor-generated time series.Time-series analysis may also require exporting data to tools such as R or SPSS, increasing pipeline complexity.
  • Scope: A TSMS is defined as a system developed or extended for storing and querying data in the form of time series.The literature also uses “Time Series Database” as another common name.
  • Prior research: Earlier TSMS research produced expressive query languages and efficient processing engines, but those systems generally lacked parallel and distributed query processing.The survey also identifies limited use-case-specific optimization in earlier general-purpose systems.
  • Survey approach: The survey analyzes current TSMS state of the art, discusses system limitations and proposed research directions, and presents a vision for a next-generation TSMS.Its search used Google Scholar, iterative structured searches, references, newer citing publications, publication venues, and author publication histories.

2 CLASSIFICATION CRITERIA

The survey classifies TSMSs by how their storage and processing components are connected, then summarizes their classification and functionality using two tables.

  • Architecture: The surveyed systems are organized into three architectural categories based on how data processing and storage components are connected.The architecture decision is treated as having a major impact on system implementation.
  • Table 1: Table 1 summarizes surveyed systems using classification criteria, with sections organized by architecture.The table identifies systems by criteria including year, implementation, and primary interaction methods.
  • Table 2: Table 2 summarizes supported query functionality using mostly SQL terminology and additional columns for append-only updates and data analytics.The surveyed functionality includes selecting data points based on time-related conditions and other operations described uniformly across systems.
  • System descriptions: The survey describes each system through its architecture, motivational use case, contributions, classification criteria, and supporting illustrations.The figures are redrawn architecture diagrams or method illustrations from the original publications.

3 INTERNAL DATA STORES

Internal-data-store TSMSs integrate storage and processing closely, enabling specialized layouts but generally yielding centralized systems or proof-of-concept implementations. The surveyed systems span AQP, wearable IoT, indexing, flash storage, and versioned analytics use cases.

  • Architecture: Integrating storage and processing in one executable enables extensive data-layout optimization and avoids a transfer protocol between components.The design also simplifies communication, although storage is not accessible to other applications.
  • Limitations: Internal-store systems include specialized constraints and maturity boundaries: tsdb requires regular time series, while several systems are incomplete or intended mainly as demonstrations.The survey states that tsdb does not support stream processing, aggregation, or AQP, and that some implementations lack deployment-level robustness.
  • Approximate querying: FAQ, RINSE, and model-based systems use sketches, adaptive indexes, or fitted models to support approximate or efficient time-series queries.RINSE’s ADS+ index supports both approximate queries and exact queries through the same data structure.
  • WearDrive: WearDrive transfers wearable sensor data to a smartphone over a wireless connection and uses volatile memory to improve wearable-device performance and battery life.Its firmware extension provides persistence guarantees comparable to non-volatile flash storage.
  • Versioned analytics: PhilDB separates data points from updates so existing readings are preserved and previous time-series versions remain analyzable.Its in-memory representation uses Pandas, connecting the system directly to the Python data-science ecosystem.
  • Discussion: Only WearDrive is distributed among the internal-store systems, while the remaining systems are primarily centralized or proof-of-concept implementations.The survey identifies tsdb, Plato, and PhilDB as the only general-purpose internal-store systems complete enough for that role.

4 EXTERNAL DATA STORES

TSMSs using external data stores combine specialized time-series processing with existing databases or distributed file systems. These systems span centralized and distributed architectures, supporting varied storage, querying, stream-processing, and approximate-query capabilities.

  • Overview: External-store TSMSs build processing components on existing DBMSs or distributed file systems, reusing deployments while adding configuration and deployment complexity.This approach reduces development effort but constrains storage models and requires multiple systems to be deployed and configured.
  • Systems: Systems provide diverse architectures and functions, including SciDB’s array-based storage and sampling, TSDB’s centralized binary caching, and Tristan’s layered compression and AQP.Tristan separates acquisition and compression, storage, and query execution, while its dictionary is trained offline and adjusted during operation.
  • Systems: Other systems emphasize specialized ingestion and analytics: Williams et al. use an in-memory data grid, Bolt supports sample-based AQP, and servIoTicy integrates user-defined stream processing.Bolt indexes recent data in memory and spills larger indexes to disk, making historical queries more expensive.
  • Systems: Gorilla and BTrDB optimize high-throughput or high-resolution storage, but their native stream-processing capabilities are limited or absent.Gorilla provides no direct stream processing, while BTrDB’s read/write performance was used by the separate DISTIL framework.
  • Systems: Respawn and Mickulicz et al. precompute multiresolution or hierarchical aggregates to support low-latency approximate range and aggregate queries.Mickulicz et al. define storage by time interval and error bound rather than raw-data size.
  • Overview: Distributed systems in this category target IoT, industrial or IT monitoring, and scientific data, generally scaling through distributed computing and in-memory caching.TSDS and Tristan are centralized exceptions that limit scalability.

5 RDBMS EXTENSIONS

This section surveys RDBMS extensions that add time-series storage, querying, forecasting, or model-based approximate query processing. The systems differ in use case and AQP scope, but none supports stream processing.

  • RDBMS extensions simplify time-series analysis by integrating storage and analysis within one system.This avoids exporting data to separate tools such as R or SPSS.
  • TimeTravel extends PostgreSQL with model-based AQP, forecasts, and a uniform SQL interface for exact historical and approximate historical or future queries.Its model hierarchy supports accuracy-based model selection and seasonality specification.
  • F2DB extends PostgreSQL for data-warehouse forecasting, supporting manually specified models and automatic selection based on accuracy and performance cost.
  • Bakkalian et al.'s Oracle extension represents time series as linear functions in an OLAP data model to reduce storage requirements and query response time.Raw events and interval models are stored in separate tables.
  • TimeTravel and F2DB provide forecasting capabilities, while Bakkalian et al. use models for OLAP storage and query efficiency rather than forecasting.TimeTravel supports AQP for historical and forecast queries; F2DB focuses on forecast queries.
  • Despite their AQP functionality, the surveyed RDBMS extensions do not implement stream-processing functionality.

6 FUTURE RESEARCH DIRECTIONS

This section reviews research directions for scaling TSMSs to heterogeneous, high-volume sensor data and outlines a distributed, model-based sensor-data warehouse. Proposed priorities include horizontal scalability, adaptive representations, parallel execution, online AQP, and continuously maintained multidimensional analytics.

  • Research Directions Proposed in Literature: Sensor-network and cloud applications require TSMSs to address heterogeneous data sources, data structuring, and scalability for large sensor-data volumes.Proposed approaches combine RDBMS and NoSQL methods, emphasize horizontal partitioning, and reduce reliance on joins.
  • Research Directions Proposed in Literature: Future TSMS research includes distributed processing, approximate query processing, and analytical methods robust to imprecise values and outliers.
  • Research Directions Proposed in Literature: Specialized systems should select storage and execution strategies from data characteristics, using adaptive representations, indexes, cost-based optimization, and parallel distributed execution.
  • Research Directions Proposed in Literature: Large-scale CPS analysis still requires methods to infer how system changes affect sensor readings before automated control can be realized.Graph-based methods and discrete-state changes are proposed as possible approaches.
  • Online Model-Based Sensor Warehouse: The envisioned next-generation TSMS is a distributed sensor-data warehouse with mathematical-model storage, real-time updates, and online distributed model-based AQP.
  • Online Model-Based Sensor Warehouse: Model-based AQP can offer configurable error bounds, lower storage requirements, missing-value inference, outlier removal, and faster answers for queries answerable from models.
  • Online Model-Based Sensor Warehouse: A future system should evaluate models for distributed stream processing, where reduced bandwidth use may support new algorithms, while current proposals have limited latency and declarative-query support.
  • Online Model-Based Sensor Warehouse: Integrating model-based data-cube representations with continuously maintained OLAP cubes remains an unresolved challenge in distributed systems.Data cubes could provide analysts with a familiar interface for multidimensional time-series analytics.

7 CONCLUSION

The survey finds architectural and capability gaps across TSMSs and proposes a distributed, time-series-specific system combining interactive analytics, stream processing, AQP, and user-defined models.

  • Conclusion: The survey provides a systematic classification of TSMSs and presents a next-generation vision based on the survey analysis and prior research directions.
  • Survey conclusions: Only a limited number of systems support stream processing during ingestion, while approximation methods are more common for reducing response time or storage requirements.Some systems instead pipe stored data through transformations or structure APIs around streams.
  • Survey conclusions: Approximation interfaces rarely let domain experts add user-defined methods or models optimized for particular domains or datasets.
  • Future research directions: Future research emphasizes TSMSs built from the ground up for time series rather than reusing general-purpose storage components or data models.Proposed systems should combine in-memory, parallel, and distributed processing to support interactive analytics and visualization.
  • Future research directions: The authors propose a distributed TSMS with data-warehouse-like analytical capabilities, real-time updates, user-defined stream processing, and interactive queries over historical and incoming data.Approximate Query Processing is proposed as the mechanism for interactive query speed.
Loading 1710.01077v1…