Multi-Scale Representation Learning for Spatial Feature Distributions using Grid Cells

Gengchen Mai; Krzysztof Janowicz; Bo Yan; Rui Zhu; Ling Cai; Ni Lao

Source-linked AI summary

Multi-Scale Representation Learning for Spatial Feature Distributions using Grid Cells

Gengchen Mai, Krzysztof Janowicz, Bo Yan, Rui Zhu, Ling Cai, Ni Lao

arXiv:2003.00824v1Feb 16, 2020cs.CVcs.AIcs.LGstat.ML

TL;DR

Spatial representation models must preserve absolute position and spatial context while handling distributions with different characteristics. Space2Vec addresses this gap with grid-cell-inspired multi-scale encodings and context-aware decoders. Across POI and image-classification tasks, it outperforms established spatial encoders and handles distributions across scales more effectively.

Problem
Existing spatial models lack a general representation that preserves geographic coordinates while jointly modeling distributions with different characteristics.
Method
Space2Vec uses an encoder-decoder framework with sinusoidal multi-scale location encodings and attention over spatially related context points.
Results
Space2Vec outperforms RBF kernels, feed-forward networks, and tile embeddings across POI-type prediction and geo-location-assisted image classification.
Takeaways & Limitations
Its multi-scale representations handle spatial distributions at different scales, whereas baselines generally perform well at no more than one scale.
Takeaways & Limitations
The POI-type grouping analysis is complicated because one POI can have multiple types belonging to different distribution groups.

Abstract

from arXiv · show

Unsupervised text encoding models have recently fueled substantial progress in NLP. The key idea is to use neural networks to convert words in texts to vector space representations based on word positions in a sentence and their contexts, which are suitable for end-to-end training of downstream tasks. We see a strikingly similar situation in spatial analysis, which focuses on incorporating both absolute positions and spatial contexts of geographic objects such as POIs into models. A general-purpose representation model for space is valuable for a multitude of tasks. However, no such general model exists to date beyond simply applying discretization or feed-forward nets to coordinates, and little effort has been put into jointly modeling distributions with vastly different characteristics, which commonly emerges from GIS data. Meanwhile, Nobel Prize-winning Neuroscience research shows that grid cells in mammals provide a multi-scale periodic representation that functions as a metric for location encoding and is critical for recognizing places and for path-integration. Therefore, we propose a representation learning model called Space2Vec to encode the absolute positions and spatial relationships of places. We conduct experiments on two real-world geographic data for two different tasks: 1) predicting types of POIs given their positions and context, 2) image classification leveraging their geo-locations. Results show that because of its multi-scale representations, Space2Vec outperforms well-established ML approaches such as RBF kernels, multi-layer feed-forward nets, and tile embedding approaches for location modeling and image classification tasks. Detailed analysis shows that all baselines can at most well handle distribution at one scale but show poor performances in other scales. In contrast, Space2Vec's multi-scale representation can handle distributions at different scales.

1 INTRODUCTION

Space2Vec addresses the lack of a general spatial representation model that preserves geographic position and context while accommodating distributions at different scales. Inspired by biological grid cells, it uses multi-scale spatial encodings and improves location modeling and image classification over established alternatives.

Motivation: Existing spatial models often discard geographic coordinates or reduce spatial structure to discretization, graphs, or feed-forward coordinate mappings.These approaches may lose cardinal direction, spatial layout, or general-purpose coordinate information.
Motivation: A central challenge is jointly modeling coexisting spatial distributions that differ substantially, such as clustered women’s clothing POIs and regularly distributed education POIs.Different POI types can exhibit distinct Ripley’s K patterns across spatial scales.
Motivation: Biological grid cells motivate multi-scale periodic location representations that can serve as metrics for encoding positions and spatial relationships.The paper connects these representations to Fourier-like combinations of cosine gratings and to continuous geographic spaces.
Approach: Space2Vec combines sinusoidal encodings of absolute positions and spatial contexts in an encoder-decoder framework with context-point multi-head attention.The attention mechanism explicitly models spatial relationships between a query point and its context points.
Evaluation: Experiments cover POI-type prediction and geo-location-assisted image classification, where Space2Vec outperforms RBF kernels, feed-forward networks, and tile embeddings.The model also visualizes encoding responses to examine how different scales capture spatial structures and distance effects.

2 PROBLEM FORMULATION

The paper formulates distributed point-feature representation as a function that maps coordinates in continuous space to vector embeddings, while associating each point with location and attributes.

Problem formulation: A spatial representation function maps any coordinate x in an L-dimensional space to a d-dimensional vector, with L smaller than d.The formulation considers L=2 or 3 and represents points of interest in geographic space.
Problem formulation: Each point combines a location coordinate with attributes such as type, name, capacity, or other feature values.The point is represented as p_i=(x_i,v_i), where x_i is location and v_i contains attributes.

3 RELATED WORK

Prior work encodes spatial information through RBF kernels, discretization, or feed-forward networks, but these approaches face limitations involving examples, scale selection, memory, and spatial structure.

Grid-cell connections: Navigation research links grid-like response patterns in trained networks to spatial localization and vector-based path integration.These findings provide context for using grid-cell-inspired representations in spatial encoding.
Existing spatial encoders: RBF kernels generate learning-friendly spatial representations but are example-based, using training-example positions as Gaussian-kernel centers.The resulting representation stores the locations of training examples.
Existing spatial encoders: Discretization-based location models require choosing a spatial partition, and incorrect choices can affect performance while increasing memory demands.The cited work identifies discretization selection and scalability as challenges.
Existing spatial encoders: Inductive location encoders avoid storing examples by directly mapping coordinates with feed-forward or residual neural networks.The related approaches cited in the paper feed coordinates into neural architectures rather than using explicit multi-scale periodic encodings.

4 METHOD

Space2Vec uses an encoder-decoder architecture that separately embeds locations and point features, reconstructing features from locations or nearby spatial context. Its location encoder uses periodic multi-scale representations, while context decoding incorporates spatial relations through attention.

Encoder-decoder framework: The framework encodes each point’s location and attributes separately, concatenating their embeddings into a full point representation.Entities without known attributes can still receive location embeddings.
Encoder-decoder framework: A location decoder reconstructs a point’s feature embedding from its location embedding, while a spatial-context decoder reconstructs it from nearby points.The two decoders can be used independently or jointly.
Point space encoder: The point-space encoder builds location representations from periodic sine and cosine functions inspired by grid-cell patterns.A single-scale theoretical representation produces hexagonal firing patterns but is not globally unique because of periodicity.
Point space encoder: Multi-scale location encodings concatenate representations across grid scales before passing them through neural layers.The formulation defines scale-dependent components using oriented vectors and sinusoidal functions.
Spatial context decoder: Spatial context decoding represents each neighbor relative to the center point using displacement vectors that preserve distance and direction.The model processes multiple context points simultaneously and uses attention weights over their embeddings.
Training: Unsupervised training maximizes the likelihood of the true point among candidate points using reconstructed and real feature embeddings.Negative sampling can improve training efficiency, while location embeddings are excluded from the candidate feature representation.

5 EXPERIMENT

Experiments compare Space2Vec with coordinate encoders and specialized context models across location modeling, spatial context modeling, and image classification. The results indicate that multi-scale grid-cell representations handle heterogeneous spatial distributions more consistently than single-scale baselines.

Experimental Setup: Experiments compare direct, tile, wrap, and RBF position encoders with Space2Vec on POI and spatial prediction tasks.The evaluation uses geographic datasets containing POI positions and types, with NLL, MRR, and HIT@5 metrics for POI prediction.
Location Modeling Evaluation: The best location-modeling results come from grid-cell models, whereas direct and theorydiag only beat random selection and single-scale tile, wrap, and RBF models perform better but remain less competitive.The comparison uses validation-selected hyperparameters and reports average metrics on the evaluation datasets.
Location Modeling Evaluation: Theory improves over RBF by -0.2%, +0.6%, and +2.1% MRR at λmin=1k, 500, and 50, respectively.At large grid sizes, theory resembles RBF; access to smaller grid sizes produces the largest reported relative improvement.
Multi-Scale Analysis of Location Modeling: Across clustered, middle, and even POI distributions, single-scale baselines trade off performance across scales, while Space2Vec handles distributions at different scales.The groups are defined by radius r as clustered (r ≤ 100m), middle (100m < r < 200m), and even (r ≥ 200m).
Spatial Context Modeling Evaluation: Grid-cell approaches outperform specialized context approaches on the test dataset and remain competitive on validation, although gains are small when context information is available.The context decoder uses nearest context points, with n=10, and compares baseline and specialized models.
Spatial Context Modeling Evaluation: Representation analysis links model behavior to spatial distance: theory integrates multiple scales, while scaled RBF uses a kernel whose size grows with distance.Theory and scaled RBF show similar polar-space representations and similar context-modeling performance.

6 CONCLUSION

The paper introduces Space2Vec as a general-purpose, unsupervised encoder-decoder model for spatial representation, inspired by multi-scale periodic grid-cell representations. Across two POI-related experiments, its ability to integrate multiple scales distinguishes it from baseline methods.

Space2Vec is an encoder-decoder framework for general-purpose spatial representation, inspired by biological grid cells’ multi-scale periodic representations.
The model is inductive and can be trained in an unsupervised manner.
Experiments evaluate POI type prediction from POI locations and from nearby POIs.
The evaluation results demonstrate the effectiveness of the model on these two tasks.
Analysis attributes the grid-cell models’ advantage to integrating representations at different scales.

A.1 BASELINES

The baselines compare direct neural encoders, grid and tile embeddings, periodic encoders, RBF features, and spatial-context-specific variants. Some context baselines additionally represent displacement in polar coordinates or rescale kernels by distance.

Direct encoders map locations or displacements to embeddings with feed-forward neural networks without multi-scale periodic decomposition.
Tile encoders divide the study area or context range into equal-sized grids and assign each grid an embedding.
Wrap normalizes coordinates, applies sinusoidal and cosine coordinate wrapping, then processes them through fully connected residual blocks.
RBF encoders use randomly sampled anchor points to construct Gaussian-kernel feature vectors that feed into a neural network.
Grid-cell comparisons include grid, hexa, theory, and theorydiag encoders, with theorydiag constraining each scale to a separate block.
Context-specific baselines include polar, polar_tile, and scaled_rbf variants designed to model spatial relationships or multiple scales.

A.2 HYPER-PARAMETER SELECTION

All methods are tuned by grid search on validation performance. The selected grid-cell configurations use different minimum scales for location modeling and spatial context modeling.

All methods use grid search based on validation-set performance.
Location Modeling: For location modeling, the best grid-cell models use d_pv = 64, d_x = 64, S = 64, and λ_min = 50.
Location Modeling: For location modeling, λ_max is fixed at 40k according to the total study-area size.
Spatial Context Modeling: For spatial context modeling, the best grid-cell models use d_pv = 64, d_x = 64, S = 64, and λ_min = 10.
Spatial Context Modeling: For spatial context modeling, λ_max is set to 10k based on the maximum center-to-context displacement.

A.3 FIRING PATTERN FOR THE NEURONS

Figure 4 presents firing patterns for the first eight of 64 neurons under different location encoders. The figure is used to compare how encoders represent spatial structure.

Figure 4 shows firing patterns for the first 8 neurons out of 64 under different encoders in location modeling.

A.4 EMBEDDING CLUSTERING OF RBF AND THEORY MODELS

Figures 5 and 6 compare embedding clustering for theory and RBF models across original-space and polar-distance-space views under varied model parameters.

Theory models: Figure 5 compares theory-model embedding clustering across different λmin values while holding λmax at 10k and S at 64.Panels (a)–(d) show original-space results, while panels (e)–(h) show polar-distance-space results.
Shared settings: All models in both figures use one hidden ReLU layer with 512 neurons; the RBF models additionally use σ=10m as the basic kernel size.
RBF models: Figure 6 compares RBF-model embedding clustering across different kernel rescalar factors β, including β=0.0 for the original RBF model.Panels (a)–(d) use the original space, and panels (e)–(h) use the polar-distance space.