Source-linked AI summary
Constructing Gazetteers from Volunteered Big Geo-Data Based on Hadoop
Song Gao, Linna Li, Wenwen Li, Krzysztof Janowicz, Yue Zhang
TL;DR
Authoritative gazetteers are costly to maintain, motivating data-driven enrichment from volunteered geographic information. The paper develops a Hadoop-based distributed platform and geoprocessing workflow for crowd-sourced gazetteer construction. Its spatial-join examples show higher efficiency than traditional desktop PC-based analysis, while the authors identify further work on conflation and quality-related research.
Problem
Authoritative gazetteers require lengthy expert efforts and high maintenance costs, while processing volunteered Big Geo-Data is computationally intensive.
Method
The paper designs a Hadoop-based distributed platform for harvesting and processing crowd-sourced gazetteer entries from volunteered geographic information.
Results
The spatial-join examples show that the MapReduce-based algorithm processes increasing point datasets more efficiently than traditional desktop PC-based analysis.
Takeaways & Limitations
The platform demonstrates the promise of using VGI and cloud computing for mining large volumes of geospatial data and connecting GIS with cloud computing.
Takeaways & Limitations
Further work is required on gazetteer conflation and on efficiency and quality research beyond names, footprints, and types.
Abstract
from arXiv · showhide
Traditional gazetteers are built and maintained by authoritative mapping agencies. In the age of Big Data, it is possible to construct gazetteers in a data-driven approach by mining rich volunteered geographic information (VGI) from the Web. In this research, we build a scalable distributed platform and a high-performance geoprocessing workflow based on the Hadoop ecosystem to harvest crowd-sourced gazetteer entries. Using experiments based on geotagged datasets in Flickr, we find that the MapReduce-based workflow running on the spatially enabled Hadoop cluster can reduce the processing time compared with traditional desktop-based operations by an order of magnitude. We demonstrate how to use such a novel spatial-computing infrastructure to facilitate gazetteer research. In addition, we introduce a provenance-based trust model for quality assurance. This work offers new insights on enriching future gazetteers with the use of Hadoop clusters, and makes contributions in connecting GIS to the cloud computing environment for the next frontier of Big Geo-Data analytics.
1. Introduction
Traditional gazetteers require lengthy expert effort and costly maintenance, while volunteered Web data offers a bottom-up alternative whose scale demands distributed processing. The paper presents a Hadoop-based platform and demonstrations for harvesting and analyzing crowd-sourced gazetteer entries.
- Authoritative gazetteers require expert teams, lengthy efforts, and high maintenance costs.
- Web documents and social-media sources can support bottom-up construction or enrichment of gazetteers.
- Big Data’s volume, velocity, and semantic and syntactic variety make existing processing tools inadequate for large geotagged streams.
- The paper presents a scalable distributed Hadoop platform for harvesting crowd-sourced gazetteer entries and conducting spatial analysis in a cloud-computing environment.
- Demonstrations extract multiple gazetteer feature types at multiple scales and show how emerging data and technologies can support GIScience research.
- The workflow, experiments, results, and trust evaluation are organized around mining, storing, analyzing, and visualizing crowd-sourced gazetteer entries.
2. Related work
Related work frames gazetteers as links between place names and geographic locations, while Web-based volunteered information expands the available descriptions. It also identifies representational, interoperability, quality-control, and large-scale processing challenges that motivate cloud-based approaches.
- Space and place: Place descriptions from social tagging, tweets, photos, and geo-social check-ins provide abundant Web-based information about places.
- Gazetteers: Existing GIS and spatial databases represent space maturely but remain limited in gazetteer services, semantic interoperability, and semi-automated feature annotation.
- Gazetteers: Gazetteers connect place names with location coordinates and commonly represent names, feature types, and footprints.
- Gazetteers: Official gazetteers may omit everyday places, emerging popular-culture names, and vague spatial extents because updates are time-consuming and policy-oriented.
- Big Data and VGI: VGI includes user-generated geospatial content in varied formats, creating opportunities to complement authoritative gazetteers with volunteered data.
- Big Data and VGI: Cloud computing offers scalable paradigms for Big Data processing, but crowd-sourced Big Geo-Data had not been thoroughly utilized in gazetteer research.
3. The Hadoop-based processing platform
The Hadoop-based processing platform combines web harvesting, distributed storage and computation, spatial extensions, cluster management, and GIS visualization to process volunteered geographic data. Its MapReduce geoprocessing workflows support scalable gazetteer analysis, while integration with GIS tools addresses raw Hadoop’s limited statistics and visualization capabilities.
- Platform architecture: The platform provides a scalable, reliable, distributed environment for mining, storing, analyzing, and visualizing gazetteer entries.Its architecture includes a web crawler, Hadoop cluster, Cloudera Manager interface, and GIS client.
- GIS integration: Raw Hadoop lacks powerful statistics and visualization tools, so the platform integrates GIS components and Esri geometry libraries.The integration spatially enables Hadoop, supports parallel gazetteer operations such as spatial filtering and joins, and links results to ArcGIS.
- Data acquisition: The web crawler harvests textual place descriptions and semi-structured geotagged data from sources such as Twitter and Flickr.Pre-processing, including removal of invalid coordinates, is required before analysis.
- Distributed processing: HDFS and MapReduce distribute storage and computation across Hadoop cluster nodes for large web datasets.MapReduce divides tasks into map and reduce subprocesses, distributing intermediate key/value pairs before merging results by key.
- Distributed processing: MapReduce accelerates text-based gazetteer analysis by distributing co-occurrence counting across nodes and combining the results.The workflow identifies the most popular words used to describe a place after parallel calculation.
- GIS integration: The GIS client visualizes MapReduce results and converts points, polylines, and polygons into Hadoop-supported formats for further spatial operations.GeoJSON stores spatial and attribute information in plain text and meets HDFS requirements.
- Workflow outcomes: The geoprocessing workflow enables fast processing and statistics of gazetteer entries while supporting other computationally intensive spatial analyses.The workflow connects ArcGIS tools with Hadoop-based distributed processing.
4. Experiments and Results
The experiments use Flickr geotagged data and a Hadoop-based workflow to extract feature-type gazetteer entries, assign them to administrative units, and compare spatial-join performance with a desktop PC. The workflow supports multi-scale spatial analysis and substantially reduces computation time for sufficiently large point sets.
- Workflow and extracted place types: The Hadoop ecosystem used distributed servers, HDFS, and MapReduce to extract prominent place types and perform spatial joins for gazetteer analysis.The experiments extracted parks, schools, museums, coffee shops, streets, and rivers, then visualized their geographic footprints.
- Datasets and Hadoop cluster: 5,319,623 Flickr geotagged records within the contiguous US were collected, with locations derived from camera GPS or user georeferencing.Metadata included photo identifiers, titles, descriptions, tags, timestamps, coordinates, and user lineage information.
- Multi-scale spatial analysis: Spatial joins assigned gazetteer entries to hierarchical geopolitical divisions including states, counties, ZIP code regions, and census tracts.The workflow also supported named-entity recognition for extracting place entities and produced operational summaries of the results.
- Gazetteer construction and constraints: The workflow demonstrated how MapReduce and GIS tools for Hadoop can derive feature-type-based gazetteer entries inside administrative polygons, while provenance information can support validation.The study notes that higher efficiency might result from adding data nodes, but Hadoop systems can encounter disk or processing bottlenecks and cluster optimization was outside the paper’s scope.
5. Conclusions and Future work
The study demonstrates a Hadoop-based approach for constructing crowd-sourced gazetteers from volunteered geographic data, combining scalable geoprocessing with place characterization and trust evaluation. It reports higher efficiency than traditional desktop analysis while identifying broader data, schema, scalability, and quality-assurance needs for future work.
- Contributions: The study builds a Hadoop-based platform for mining, analyzing, and visualizing crowd-sourced gazetteer entries from volunteered geographic data.The platform is presented as a way to connect GIS and cloud computing for Big Geo-Data analytics.
- Contributions: Three geographic feature types—points, polylines, and polygons—are extracted, analyzed, and visualized in the study.
- Quality assurance: The study introduces a reputation model for trust evaluation and describes semi-automatic crowd-sourced gazetteer construction supported by high-performance computing.
- Results: The MapReduce-based analysis processes increasing numbers of points more efficiently than traditional desktop PC-based analysis.The comparison is demonstrated through spatial-join operations across different geographic scales.
- Results: MapReduce counting of co-occurrence words rapidly extracts place semantics and popular tags for characterizing places.
- Contributions: The platform supports scalable geoprocessing workflows using Hadoop and Esri GIS tools, extending GIS into a cloud-computing environment.
- Future work: Future work includes conflating more sources, expanding Hadoop-supported spatial functions, extending gazetteer schemas, and studying efficiency and quality assurance.The current research uses only two MapReduce algorithms and 10 connected server-nodes for Flickr geotagged data.
- Future work: Further evaluation is needed for clusters with hundreds of nodes and more heterogeneous volunteered data sources.