PaperlayerView exact revision on arXiv

Source-linked AI summary

Geospatial Big Data Handling Theory and Methods: A Review and Research Challenges

S. Li, S. Dragicevic, F. Anton, M. Sester, S. Winter, A. Coltekin, C. Pettit, B. Jiang, J. Haworth, A. Stein, T. Cheng

arXiv:1511.03010v1physics.soc-phcs.CYphysics.data-an

TL;DR

Geospatial big data challenges the adequacy of traditional methods because its massive volume, varied formats, and near-real-time streams complicate handling and quality assessment. This paper reviews existing theory and methods, synthesizes current challenges, and identifies priority areas for further development, concluding that big data offers substantial opportunities for societal management and decision-making as research matures.

  • Problem

    Traditional geospatial data-handling approaches are inadequate for massive, varied datasets that challenge storage, processing, analysis, visualization, and quality assessment.

  • Method

    The paper reviews geospatial theory and methods developed for traditional data, assesses their applicability to geospatial big data, and synthesizes research challenges and development priorities.

  • Results

    The paper identifies priority needs including spatial indexing and algorithms for real-time streaming data, methods for causal and explanatory analysis, and improved multidimensional visualization.

  • Takeaways & Limitations

    Big data presents both challenges and opportunities, with substantial potential for societal management and decision-making as research matures.

  • Takeaways & Limitations

    Geospatial big data are often collected ad hoc and repurposed, making data veracity a substantial challenge for scalable, timely analysis.

Abstract

from arXiv · show

Big data has now become a strong focus of global interest that is increasingly attracting the attention of academia, industry, government and other organizations. Big data can be situated in the disciplinary area of traditional geospatial data handling theory and methods. The increasing volume and varying format of collected geospatial big data presents challenges in storing, managing, processing, analyzing, visualizing and verifying the quality of data. This has implications for the quality of decisions made with big data. Consequently, this position paper of the International Society for Photogrammetry and Remote Sensing (ISPRS) Technical Commission II (TC II) revisits the existing geospatial data handling methods and theories to determine if they are still capable of handling emerging geospatial big data. Further, the paper synthesises problems, major issues and challenges with current developments as well as recommending what needs to be developed further in the near future. Keywords: Big data, Geospatial, Data handling, Analytics, Spatial Modeling, Review

1. Introduction

Geospatial big data is expanding rapidly in volume, variety, and speed, challenging traditional data-handling technologies and motivating a review of current methods and future research needs.

  • Motivation: Geospatial big data comes from sensors, messaging systems, social networks, and traditional measurement systems, creating opportunities across societal applications.Examples include climate change, disease surveillance, disaster response, infrastructure monitoring, and transportation.
  • Defining big data: Big data is commonly characterized by massive structured and unstructured datasets that traditional technologies cannot easily capture, store, manage, analyze, or present.The 3Vs are Volume, Velocity, and Variety; Veracity was added to describe data integrity and quality.
  • Geospatial characteristics: Geospatial data combines geographic references with diverse sources including surveying, photogrammetry, remote sensing, laser scanning, sensors, GNSS tracking, and geo-tagged content.These sources produce data exhibiting at least one of the 3Vs and often additional characteristics.
  • Geospatial characteristics: Petabyte archives, real-time observations, social media, VGI, imagery, maps, text, raster, and vector data create storage, analysis, modeling, indexing, and management challenges.Volume increases storage and analysis demands, while variety requires more efficient data models and management strategies.
  • Paper objectives: The paper revisits existing theories and methods, examines state-of-the-art developments, synthesizes challenges, and recommends near-future research directions.Its scope covers modeling, processing, analyzing, and visualizing geospatial big data while excluding several related image-processing topics.

2. Collection of Geospatial Big Data

New sensors, everyday devices, public contributors, and digital services are transforming geospatial collection from sparse, authoritative capture toward abundant, heterogeneous, real-time data.

  • Emerging sources: New sensors and stakeholders have increased the availability of spatial information through VGI, geo-sensor networks, and user-centric data sources.These sources extend authoritative topographic datasets through greater thematic variation and user participation.
  • Collection paradigm: Geospatial data collection is shifting from technically demanding, accurate, and expensive devices toward commodity capture embedded in everyday devices.This shift enables unprecedented volumes of environmental geospatial information.
  • Sensor configurations: New sensor configurations include moving objects that capture trajectories and environments and static sensors that continuously observe changing surroundings.The resulting data can be streams with high density, multiple measured dimensions, and variable positional accuracy.
  • Application sources: Smart-city systems and social media generate vast real-time datasets from ticketing, vehicle tracking, CCTV, toll systems, traffic sensors, and geotagged posts.These sources can support applications such as disaster management and emergency response.
  • Data quality challenges: Health data may have poor spatial and temporal quality because residence location and facility-visit time do not necessarily match disease location and incidence time.This illustrates how the meaning of recorded coordinates and timestamps can diverge from the phenomenon being studied.
  • VGI: VGI may be participatory, as in OpenStreetMap, or opportunistic, when users contribute information unconsciously or for another purpose.The distinction concerns whether users intentionally select and contribute objects and their features.
  • VGI: VGI is useful when ad-hoc information is needed during disasters, although its accuracy may be limited.Social media streams have been evaluated as information sources for detecting flooding and supporting crisis management.
  • Data quality challenges: New geoscientific sources produce diverse, inaccurate, incomplete, or erroneous data that gain relevance through careful integration and fusion with complementary data.The value of these collections therefore depends on combining heterogeneous sources appropriately.

3. Quality Assessment

Geospatial big data intensifies uncertainty, asynchronicity, incompleteness, and variable quality, while traditional quality frameworks struggle to support near-real-time decisions from ad-hoc data streams.

  • Quality characteristics: Geospatial big data is voluminous, heterogeneous, real-time, inconsistent, and variable in quality, increasing uncertainty, asynchronicity, and incompleteness.These characteristics arise because geospatial data abstract and observe a continuous reality.
  • Quality frameworks: Existing quality frameworks describe and propagate characteristics quantitatively and qualitatively, translating producer descriptions into consumer fitness for purpose.Such frameworks have been formalized in international standards and related methods.
  • Decision support: Quality descriptions stored as metadata require further analysis to determine how data properties affect decisions, but decision-making remains a complex black box.Quantitative descriptors may support error propagation, whereas qualitative descriptors often remain dependent on human judgment.
  • Near-real-time assessment: Traditional metadata collection, aggregation, propagation, and human judgment fall short when geospatial data streams are combined ad hoc for near-real-time analytics.Connected urban transport illustrates how bandwidth constraints can force reduced sampling or decentralized computation, affecting quality assessment.
  • VGI quality: OpenStreetMap quality research has focused mainly on completeness, reflecting the importance of evaluating volunteered geographic information for navigation services.OpenStreetMap has become a global open source for navigation.
  • Analytics and quality: Data mining and machine learning provide inductive reasoning through correlations and predictions, but analytical correlations can replace traditional provider- or consumer-oriented quality assessments.Early traffic-accident discovery is given as a transport example.

4. Data Modelling and Structuring

Geospatial big data can use established spatial models, but very large and concurrent datasets require adaptive statistics, parallel and functional programming, streaming-aware processing, and efficient indexing.

  • Spatial data models: Established vector, network, topological, raster, and tessellation models can handle geospatial big data, although their suitability varies with dataset size and structure.The paper distinguishes models that are more suitable from those less suitable for very large datasets.
  • Spatial statistics: Spatial statistics can summarize big data and express variation and uncertainty, but many procedures were developed for smaller, pointwise datasets.The paper identifies the need for improved database structures and adaptive spatial statistical procedures.
  • Programming paradigms: Parallel and distributed programming, functional programming, MapReduce, data-flow graphs, and self-adjusting computation have been proposed for big-data handling.A research gap remains between general functional-programming work and geospatial work that guarantees the absence of data races.
  • Concurrency: Data races arise when concurrent threads access the same data items, while functional programming controls simultaneous access to mutable data.This provides a reason for considering functional languages in parallel geospatial big-data processing.
  • Programming paradigms: Domain-specific languages support multidisciplinary collaboration, while MapReduce uses map and reduce functions for distributed parallel processing of big datasets.These approaches offer flexibility and control for developing big-data systems.
  • Data mining: Spatial data-mining techniques differ in their assumptions, including probability distributions, local smoothness, continuity, and susceptibility to the curse of dimensionality.The paper contrasts parametric, non-parametric, functional-analysis, and homotopy-continuation approaches.
  • Stream processing: Real-time processing of geospatial streams may require tolerating soft errors or using high-performance computing architectures to exploit sufficient parallelism.The design choice depends on whether soft errors are acceptable.
  • Spatial indexing: Spatial indexing partitions search spaces into tiles and uses space-filling curves to improve access to n-dimensional geospatial data.The central challenge is organizing tiles so queries access data efficiently.

5. Data Visualization and Visual Analytics

Geospatial big-data visualization and visual analytics support exploration, access, and decision-making, but scale and human cognitive limits constrain their effectiveness. The section highlights GIS integration, geoportals, urban data infrastructures, and animated network visualizations as practical responses.

  • Visualization and visual analytics: Visual analytics combines computational tools with human visual sense-making to address limitations in both computation and human interpretation.
  • Visualization and visual analytics: Visualization is part of the analysis process for exploring geospatial big data and building hypotheses, not merely communicating results.However, large datasets can produce busy displays and information overload.
  • GIS and big data: GIS provides a mature framework for processing and visualizing spatial and attribute data, including data that are not perfectly structured.
  • Access and decision support: Geoportals make geospatial big data accessible to researchers, policymakers, decision-makers, and citizens through visualization and interrogation tools.The INSPIRE Geoportal combines a map window with a folksonomy tag cloud, while AURIN provides access to over 1,800 datasets and more than 6 billion data elements.
  • Urban applications: Urban big-data initiatives aim to help researchers, planners, policymakers, and communities visually explore data for collective city planning and decision-making.These efforts respond to rapid urbanization and the growth of real-time and historical smart-city data.
  • Network visualization: Animated Oyster card visualizations show individual and aggregated journeys through London over typical weekday time sequences, supporting analysis of commuter travel behavior.Such insights can inform transport planners when optimizing timetables and responding to events.

6. Data Mining and Knowledge Discovery

The paper reviews knowledge-discovery methods for geospatial big data and argues that fractal analysis offers a complementary way to recognize and model complex spatial structure.

  • Knowledge discovery: Geospatial big data analysis combines statistical analysis, geo-computing, simulation, and data-mining methods to discover knowledge.
  • Knowledge discovery: Knowledge discovery seeks valid, novel, useful, and understandable patterns from datasets that may violate traditional statistical assumptions.
  • Knowledge discovery: Association rule mining searches for events occurring in spatio-temporal neighborhoods, while regression and classification model relationships and assign class labels.
  • Knowledge discovery: Anomaly detection and clustering remain challenging because normality can evolve over time and spatio-temporal processes change.
  • Knowledge discovery: Machine learning addresses nonlinearity, multiscale structure, and heterogeneity, but large-sample computation and changing data distributions require online and parallel approaches.
  • Fractals emerged from big data: Big data reveal fractal structure with far more small things than large ones, motivating fractal geometry and complexity science for analytics and visualization.
  • Fractals emerged from big data: Tweet locations can generate natural cities by slicing data over time, constructing triangulated irregular networks, and merging edges smaller than the mean.
  • Fractals emerged from big data: The paper proposes a relaxed fractal definition based on recurring scaling patterns and uses head/tail breaks to classify, measure complexity, and visualize hierarchical structure.

7. Challenges, Open Issues and Future Directions

The review identifies unresolved challenges in scalable geospatial big-data representation, streaming computation, statistical analysis, visualization, and quality assessment, then outlines research directions centered on functional, distributed, and uncertainty-aware methods.

  • Efficient representation and modelling for geospatial big data: Streaming spatial algorithms must process incoming data in real time while minimizing network communication, disk input/output, and storage overhead.Traditional divide-and-conquer and line/planesweep algorithms cannot wait for all data, while incremental algorithms may not fit streams in main memory.
  • Efficient representation and modelling for geospatial big data: Functional and interval analysis are proposed for high-dimensional geospatial data and uncertainty modelling, but their adoption requires bridging machine-learning and functional-analysis communities.Wavelets avoid the curse of dimensionality, while interval analysis represents uncertainty in sensor inputs and evaluated functions.
  • Efficient representation and modelling for geospatial big data: A locally distributed stream-sensing paradigm would combine ontology-based specifications, functional analysis, pure functional programming, visualization, parallelization, concurrency, and cloud libraries.The proposed architecture distributes sensing, processing, visualization, decision updates, and communication across fractal-like local and regional nodes.
  • Efficient representation and modelling for geospatial big data: Future streaming algorithms are intended to combine compact geometric topology structures, local CPU/GPU parallelization, interval uncertainty handling, automatic triggers, and wavelet decompositions.These design choices target large-volume streams and automatic updates when input changes affect computed results, visualizations, or decisions.
  • Analysing, mining and visualising geospatial big data for decision-support: Spatial statistical procedures remain constrained by doubtful data quality, slower calculations, strong assumptions, and unavailable preprocessing tools for automatically handling big data.Relevant assumptions include normality, independence, and the absence of noise in explanatory data.
  • Analysing, mining and visualising geospatial big data for decision-support: Open issues include real-time predictive algorithms, representative and trustworthy data, scalable visualization without information overload, and quality assessment under incomplete data semantics.Sensor data may be irrelevant or repetitive, crowd-sourced data may underrepresent groups, and browser-based visualization of massive multidimensional datasets remains difficult.

8. Conclusions

The review finds traditional geospatial data-handling approaches inadequate for emerging big data and identifies technical, conceptual, visualization, and interdisciplinary priorities for further development. Privacy and security remain essential management concerns but fall outside the paper’s data-handling focus.

  • Traditional data-handling approaches are inadequate for geospatial big data and require further development and research.
  • New spatial indexes and algorithms are needed for real-time streaming data and topology-supported analytics.
  • Conceptual and methodological approaches should advance big-data research from descriptive and correlational work toward causal and explanatory relationships.
  • Efficient visualization methods must integrate three geographic dimensions with one dimension of continuous time and fit users’ tasks and capacities.
  • Spatial statistics, semantics, ontologies, and scale require further elaboration as general conceptual and practical issues.
  • Privacy and security are essential concerns in geospatial big-data management but are not covered because the paper focuses on data-handling methods.

Authors Contribution Statement

All authors contributed equally through discussion, section writing and revision, corresponding-section review, and comments on the complete manuscript.

  • All authors contributed equally to discussions, writing and revising sections, and reviewing the whole manuscript.
Loading 1511.03010v1…