PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning for Spatio-Temporal Data Mining: A Survey

Senzhang Wang, Jiannong Cao, Philip S. Yu

arXiv:1906.04928v2cs.LGstat.ML

TL;DR

The rapid growth of spatio-temporal data has exposed limitations in traditional data-mining methods. This survey synthesizes deep learning for STDM through data and representation categories, a general pipeline, task and application reviews, and discussion of limitations and future directions. More than 70% of related papers focus on prediction, with representation learning the second-largest problem category.

  • Problem

    Rapidly growing spatio-temporal datasets and their complex spatial-temporal structure motivate a review of methods for mining them.

  • Method

    The survey categorizes ST data and representations, introduces suitable deep learning models and a general pipeline, and reviews STDM tasks and applications.

  • Results

    More than 70% of related papers focus on ST data prediction, while representation learning is the second-largest problem category.

  • Takeaways & Limitations

    The survey organizes deep learning-based STDM research across data types, tasks, models, and application domains, while identifying open problems for future work.

Abstract

from arXiv · show

With the fast development of various positioning techniques such as Global Position System (GPS), mobile devices and remote sensing, spatio-temporal data has become increasingly available nowadays. Mining valuable knowledge from spatio-temporal data is critically important to many real world applications including human mobility understanding, smart transportation, urban planning, public safety, health care and environmental management. As the number, volume and resolution of spatio-temporal datasets increase rapidly, traditional data mining methods, especially statistics based methods for dealing with such data are becoming overwhelmed. Recently, with the advances of deep learning techniques, deep leaning models such as convolutional neural network (CNN) and recurrent neural network (RNN) have enjoyed considerable success in various machine learning tasks due to their powerful hierarchical feature learning ability in both spatial and temporal domains, and have been widely applied in various spatio-temporal data mining (STDM) tasks such as predictive learning, representation learning, anomaly detection and classification. In this paper, we provide a comprehensive survey on recent progress in applying deep learning techniques for STDM. We first categorize the types of spatio-temporal data and briefly introduce the popular deep learning models that are used in STDM. Then a framework is introduced to show a general pipeline of the utilization of deep learning models for STDM. Next we classify existing literatures based on the types of ST data, the data mining tasks, and the deep learning models, followed by the applications of deep learning for STDM in different domains including transportation, climate science, human mobility, location based social network, crime analysis, and neuroscience. Finally, we conclude the limitations of current research and point out future research directions.

I. INTRODUCTION

Spatio-temporal data mining is increasingly important as datasets grow, while traditional methods struggle with complex spatial-temporal structure and raw data representation. This survey reviews deep learning approaches for STDM, organizing data types, models, tasks, applications, and future directions.

  • Motivation: Growing spatio-temporal datasets support applications in climate, public safety, transportation, and human mobility, increasing the importance of STDM.Examples include wind and precipitation forecasting, crime prediction, traffic flow prediction, and human trajectory mining.
  • Motivation: Traditional STDM methods struggle with continuous spatial data, combined spatial-temporal properties, and limited ability to process raw data without feature engineering.Conventional analysis may require domain expertise to transform raw inputs such as fMRI pixel values into suitable representations.
  • Deep Learning Motivation: Deep learning models can automatically learn hierarchical representations from raw ST data, including spatial proximity and long-term temporal correlations.CNN convolutional layers and RNN recurrent structures are described as mechanisms for learning these patterns directly.
  • Survey Contributions: The survey presents a general STDM framework covering data-instance construction, data representation, model selection, and problem solving.The framework is intended to guide the choice or design of representations and deep learning models for particular STDM tasks.
  • Survey Contributions: It provides a comprehensive overview of deep learning for predictive learning, representation learning, classification, estimation, inference, anomaly detection, and other STDM problems across multiple domains.Applications are categorized across transportation, climate science, human mobility, location-based social networks, crime analysis, and neuroscience.
  • Future Directions: The survey identifies open problems and points toward future research directions in deep learning-based STDM.The paper’s organization covers data types and models, a general framework, STDM tasks, applications, and limitations.

II. CATEGORIZATION OF SPATIO-TEMPORAL DATA

The survey categorizes spatio-temporal data by how observations are collected and represented, covering events, trajectories, point references, raster data, and videos. These types differ in their spatial and temporal structure and suitability for deep-learning models.

  • Event data record discrete events at point locations and times, including crimes, disease outbreaks, traffic accidents, and social events.
  • Trajectory data represent paths traced by moving objects as sequences of locations and times, commonly collected through GPS-enabled sensors.
  • Point reference data measure continuous spatio-temporal fields at moving reference points, such as weather observations from floating sensors.
  • Raster data measure continuous or discrete fields at fixed spatial locations and time points, unlike point reference data whose locations change.
  • Videos encode sequences of images with spatial pixel correlations and temporal dependencies, but the survey excludes video analysis from its coverage.

B. Data Instances and Representations

The survey organizes spatio-temporal data into instances such as points, trajectories, time series, spatial maps, and rasters, then maps them to sequence, graph, matrix, or tensor representations. Representation choice depends on the mining task and selected deep-learning model.

  • Data instances: Spatio-temporal data instances include points, trajectories, time series, spatial maps, and spatio-temporal rasters.
  • Data instances: Trajectories can be represented as points, sequences, or matrices indicating whether a trajectory traverses grid regions.
  • Data instances: Raster measurements can be treated as time series at a grid, spatial maps at a timestamp, or the complete raster across locations and times.
  • Data representations: The four general input representations are sequences, graphs, two-dimensional matrices, and three-dimensional tensors.
  • Data representations: Spatial maps may use graphs or matrices, while raster data commonly use matrices or tensors; matrices are simpler but lose spatial correlations among locations.

C. Preliminary of Deep Learning Models

The survey introduces RBM, CNN, GraphCNN, RNN, LSTM, autoencoder, stacked autoencoder, and Seq2Seq models used for spatio-temporal data mining. These models support feature learning, spatial processing, temporal dependency modeling, reconstruction, and sequence transformation.

  • RBMs are two-layer stochastic networks used for dimensionality reduction, classification, feature learning, and collaborative filtering.
  • CNNs process image-like data through convolution, pooling, fully connected, and output layers.
  • GraphCNNs aggregate information across graph neighbors so node embeddings capture increasingly multi-hop context for node-level or graph-level prediction tasks.
  • RNNs model sequential characteristics, while LSTMs extend them with memory units and gates that learn long-term input dependencies.
  • Autoencoders learn compressed hidden representations and reconstruct inputs, whereas Seq2Seq models map fixed-length inputs to outputs whose lengths may differ.

III. FRAMEWORK

The framework presents a general pipeline for applying deep-learning models to spatio-temporal data mining. It begins with raw sensor data and proceeds through instance construction, representation, model design, and the target mining problem.

  • The pipeline contains spatio-temporal instance construction, data representation, deep-learning model selection and design, and problem solving.
  • Raw data from location sensors, including events, trajectories, point references, and rasters, are first organized into data instances such as points, time series, maps, trajectories, and rasters.

A. ST Data Preprocessing

ST data preprocessing converts spatio-temporal instances into representations suitable for deep learning models, while the same data type may admit multiple formats depending on the application.

  • Deep learning inputs may be vectors, matrices, or tensors, depending on the model and spatio-temporal data type.Trajectory and time-series data are typically represented as sequences, spatial maps as 2D matrices, and spatio-temporal rasters as 2D matrices or 3D tensors.
  • Trajectory data can be encoded as a matrix by partitioning the measured field into grid cells and marking visited cells.This representation exposes spatial structure for CNN-based processing.
  • Spatial maps may instead be represented as graphs when sensors are nodes and road segments define edges between neighboring sensors.Graph representations support GraphCNN models for predicting traffic variables across nodes.
  • Spatio-temporal raster data can use either 2D matrices or 3D tensors, depending on the data and application.

B. Deep Learning Model Selection & Design

Model selection in STDM connects data representation and task requirements: recurrent, convolutional, graph, autoencoding, and hybrid models are matched to different spatio-temporal structures and objectives.

  • Deep Learning Model Selection & Design: RNN, LSTM, and GRU process sequence data, while Seq2Seq models support simultaneous multi-step prediction.Seq2Seq traffic prediction commonly uses LSTM encoder and decoder layers to predict several future time slots.
  • Deep Learning Model Selection & Design: CNN models process image-like spatial maps and raster data, whereas GraphCNN handles graph-structured spatial data.Transportation networks can be modeled as directed graphs for graph-based traffic prediction, while spatial maps are commonly treated as matrices.
  • Deep Learning Model Selection & Design: Model selection depends on both the input data representation and the particular data-mining task.The pipeline feeds represented data into selected or designed deep learning models before addressing tasks such as prediction, classification, representation learning, or anomaly detection.
  • Deep Learning Model Selection & Design: Prediction is the largest studied STDM problem, with more than 70% of related papers focusing on ST data prediction.Representation learning is the second-largest category, followed by classification, detection, inference or estimation, and recommendation.
  • Deep Learning Model Selection & Design: CNN–RNN hybrids such as ConvLSTM capture spatial and temporal correlations in sequences of spatial maps.ConvLSTM uses convolutional structures in both input-to-state and state-to-state transitions within a sequence-to-sequence model.
  • Deep Learning Model Selection & Design: Trajectory prediction uses RNN or LSTM for location sequences and CNN for matrix representations that emphasize spatial correlations.

B. Representation Learning

Representation learning uses deep models to derive abstract, useful encodings of trajectories and spatial maps for downstream mining and machine learning tasks.

  • B. Representation Learning: Representation learning composes linear or nonlinear transformations to produce abstract representations that support downstream tasks.
  • B. Representation Learning: Trajectory representation learning uses RNN and CNN models for problems such as trajectory similarity, mobility modeling, and recommendation.Seq2Seq models learn trajectory representations for similarity computation, while RNN and GRU capture short- and long-term sequential relatedness.
  • B. Representation Learning: Joint neural models can learn social-network and mobility-trajectory representations together to model their correlations.
  • B. Representation Learning: Spatial-map representation learning extracts spatiotemporal features from raw sensor maps and combines static POIs with dynamic mobility data for urban community structures.

C. Classification

Deep learning classification in STDM is concentrated on neuroimaging, where convolutional architectures operate on functional connectivity descriptors or raw 3D MRI data.

  • C. Classification: Classification is mostly studied for fMRI data within deep-learning-based STDM research.The broader neuroimaging context also includes EEG and MEG.
  • C. Classification: The connectome-convolutional neural network combines diverse functional-connectivity metrics for connectome classification or regression.Its inputs can vary by changing the combinations of connectivity descriptors used for training.
  • C. Classification: 3D-CNN models classify structural MRI volumes directly, without intermediate handcrafted feature extraction.The surveyed architectures include modifications of plain and residual convolutional networks.

D. Estimation and Inference

Deep learning methods support estimation, inference, and anomaly detection across spatial maps, trajectories, and events. Applications include fine-grained air-quality inference, travel-time estimation, activity inference, and extreme-event detection.

  • Spatial maps: Limited monitoring-station coverage motivates deep learning for fine-grained air-quality inference at arbitrary urban locations.ADAIN models heterogeneous data using pollutant observations from available monitoring stations.
  • Trajectories: Deep models estimate travel times from mobility trajectories, either for paths or directly for entire trips.DEEPTRAVEL uses an RNN, while DeepTTE estimates whole-path travel time end to end.
  • Trajectories: Graph convolutional networks infer users’ activity types from spatio-temporal graphs constructed from GPS-derived activity areas and connectivities.Activity types correspond to trip purposes in the described application.
  • Anomaly detection: Anomaly-detection studies target non-recurring traffic congestion, traffic accidents, and extreme climate events in event and spatial-map data.The cited methods use CNNs or other deep models to identify disruptions and classify or localize climate extremes.

F. Other tasks.

Beyond core predictive and detection tasks, deep learning is applied to recommendation, pattern and relation mining, multimodal fusion, and attention-based spatio-temporal modeling. These approaches combine heterogeneous inputs or selectively weight spatial and temporal information.

  • Other tasks: Deep learning models support additional STDM tasks including POI recommendation, pattern mining, and relation mining.CAPE learns POI embeddings from text while capturing geographical influence and POI characteristics.
  • Data fusion: Multisource data fusion combines spatio-temporal data with correlated external information to jointly learn features and improve STDM performance.Relevant external factors include weather, social events, and holidays in traffic-flow applications.
  • Data fusion: Raw data-level fusion integrates multiple sources before feeding them into a deep learning model for feature learning.A ConvLSTM traffic-accident model combines gridded traffic, road, weather, and satellite-image features.
  • Data fusion: Latent feature-level fusion processes different raw features with separate deep models before combining their learned representations.ST-ResNet uses this strategy to forecast regional crowd inflow and outflow.
  • Attention: Attention mechanisms can improve STDM models by weighting relevant information in sequential data and image-like spatial maps.STDM attention is categorized into spatial, temporal, or combined spatial-temporal attention.

V. APPLICATIONS

Deep learning applications span transportation and on-demand services, where abundant urban data support forecasting and incident-related tasks. Transportation and human mobility account for the largest share of surveyed application-domain work.

  • Application scope: The surveyed literature covers transportation, on-demand service, climate and weather, human mobility, LBSN, crime analysis, and neuroscience.Table II summarizes related works across these application domains.
  • Transportation: Transportation applications use sensor-derived data to learn complex spatio-temporal traffic correlations for flow prediction, incident detection, and congestion prediction.Sources include loop detectors, road cameras, and GPS.
  • On-demand services: On-demand services generate spatio-temporal data describing customer locations and required service times, enabling demand-and-supply forecasting.Examples include Uber, DiDi, Mobike, and GoGoVan.
  • On-demand services: Forecasting methods include deep learning for dockless-bike demand-supply distributions, station-level bike demand, taxi demand, and online car-hailing supply-demand.Reported models include graph CNNs, LSTMs, and ResNet.

C. Climate & Weather

Climate and weather data contain strongly correlated atmospheric and oceanic conditions across locations and time. Deep learning therefore supports climate-related spatio-temporal analysis alongside human mobility, LBSN, and crime applications.

  • Climate and weather: Climate data comprise atmospheric and oceanic conditions collected by sensors at fixed or floating locations.Examples include temperature, pressure, wind-flow, and humidity.
  • Human mobility: Human mobility research uses geolocated datasets to model individual and collective trajectory regularities for forecasting, planning, behavior analysis, and recommendation.Deep learning applications include trajectory classification, prediction, and representation learning.
  • Location-based social networks: Location-based social networks connect users through physical locations and location-tagged content.Foursquare and Flickr are given as examples of such networks.
  • Crime analysis: Crime datasets record crime types together with event times and locations, allowing analysis of spatial-temporal crime patterns and law-enforcement policy effects.Deep learning can use city crime-account heat maps as model input.

G. Neuroscience

Neuroscience supplies diverse spatio-temporal measurements with markedly different spatial and temporal resolutions. The survey presents deep-learning STDM as broad in scope while identifying interpretability, model selection, task coverage, and multimodal fusion as unresolved challenges.

  • G. Neuroscience: Neuroimaging technologies including fMRI, EEG, MEG, and fNIRS measure neural activity at substantially different spatial and temporal resolutions.fMRI measures activity from millions of locations, whereas some other technologies measure it from far fewer locations.
  • Open problems: Deep-learning STDM models remain difficult to interpret, especially for complex spatio-temporal data representations.Attention mechanisms have improved interpretability for periodicity and local spatial dependency, but broader interpretability remains understudied.
  • Open problems: Selecting suitable ST data representations and deep-learning models remains understudied, with traffic modeled as time series, spatial maps, or graphs.These representations correspond respectively to RNN/DNN/SAE, CNN, and GraphCNN approaches.
  • Open problems: Deep learning has scarcely addressed frequent pattern mining and relationship mining because these tasks may not require learned features.The survey identifies extending deep learning alone or integrating it with traditional models as an open problem.
  • Open problems: Fusing multimodal spatio-temporal datasets remains insufficiently studied in neuroimaging and urban transportation.Examples include combining fMRI with DTI, or taxi trajectories with bike-sharing and public-transport data.
Loading 1906.04928v2…