Source-linked AI summary
Tile2Vec: Unsupervised representation learning for spatially distributed data
Neal Jean, Sherrie Wang, Anshul Samar, George Azzari, David Lobell, Stefano Ermon
TL;DR
Geospatial analysis has abundant remote sensing data but often lacks labels, while existing natural-image pre-training does not fit remote sensing’s distinctive bird’s-eye and multi-spectral characteristics. Tile2Vec learns representations from unlabeled data by applying distributional semantics to spatial neighborhoods and training a margin-based triplet-loss CNN. Across land-cover and poverty-prediction tasks, the representations outperform unsupervised alternatives and, in one classification setting, supervised CNNs trained on 50k labels.
Problem
Remote sensing provides abundant data, but many downstream tasks lack labels, and ImageNet pre-training is poorly suited to its bird’s-eye, often multi-spectral imagery.
Method
Tile2Vec trains a CNN on spatial triplets, bringing geographically neighboring tile embeddings closer while separating distant tiles with a margin-based loss.
Results
Tile2Vec outperforms other unsupervised features and even supervised CNNs trained on 50k labeled examples; Uganda poverty prediction reaches r2 = 0.496 ± 0.014 versus r2 = 0.41 for transfer learning.
Takeaways & Limitations
The method supports representation learning across aerial and satellite imagery, multiple spectral modalities, and a non-image economic-data task.
Takeaways & Limitations
The paper exploits spatial coherence but does not yet incorporate temporal patterns such as seasonality or crop cycles.
Abstract
from arXiv · showhide
Geospatial analysis lacks methods like the word vector representations and pre-trained networks that significantly boost performance across a wide range of natural language and computer vision tasks. To fill this gap, we introduce Tile2Vec, an unsupervised representation learning algorithm that extends the distributional hypothesis from natural language -- words appearing in similar contexts tend to have similar meanings -- to spatially distributed data. We demonstrate empirically that Tile2Vec learns semantically meaningful representations on three datasets. Our learned representations significantly improve performance in downstream classification tasks and, similar to word vectors, visual analogies can be obtained via simple arithmetic in the latent space.
1 Introduction
Remote sensing offers abundant data for important applications, but many downstream tasks lack the labels required by supervised learning. Tile2Vec addresses this gap by learning representations from unlabeled remote sensing data and reports strong downstream performance.
- Remote sensing supports applications including land-use monitoring and precision agriculture, but many downstream tasks are constrained by limited labels.
- Existing pre-training methods from natural images are poorly suited to remote sensing because its bird’s-eye, often multi-spectral imagery has distinct characteristics.
- Tile2Vec learns compressed, informative representations from unlabeled remote sensing data using spatial context.
- Tile2Vec generalizes across data modalities, trains stably, and remains robust to hyperparameter choices.
- The learned representations improve difficult land-use classification, exceed supervised models trained on large labeled sets, and support visual queries and analogies.
2 Tile2Vec
Tile2Vec extends distributional semantics to geospatial data by treating image tiles as atomic units and geographic neighborhoods as context. It trains a CNN with an unsupervised triplet loss, spatially samples training examples, and uses regularization and augmentation to obtain scalable, meaningful representations.
- 2.1 Distributional semantics: Tile2Vec asks which atomic units and contexts can support distributional semantics for geospatial data.
- 2.1 Distributional semantics: Remote sensing tiles serve as high-dimensional atomic units, with CNNs projecting them into lower-dimensional representations.
- 2.1 Distributional semantics: Geographic neighborhoods provide weak supervision: nearby tiles are assumed to have more similar semantics and representations than distant tiles.
- 2.2 Unsupervised triplet loss: Each training triplet contains an anchor, a geographically close neighbor, and a distant tile; the loss pulls anchor-neighbor embeddings together and separates anchor-distant embeddings.
- 2.2 Unsupervised triplet loss: A rectified margin prevents unlimited separation by stopping optimization once the distant embedding exceeds the neighbor distance by the required margin.
- 2.2 Unsupervised triplet loss: Embedding-norm regularization constrains representations to a hypersphere so that relative distances retain meaning.
- Sampling: Sampling uses tile size and neighborhood parameters, with distant tiles drawn outside the neighborhood; no knowledge of actual geographic locations is required.
- 2.4 Scalability: Data augmentation lets the training dataset scale with available remote sensing data and produces training that is easy, robust to hyperparameters, and resistant to overfitting.
3 Datasets
Tile2Vec is evaluated across remote-sensing and non-image spatial datasets, covering land-cover classification, poverty regression, and country-characteristic prediction. The experiments use multiple imagery sources, spectral resolutions, labels, and unsupervised-baseline comparisons.
- Overview: The evaluation spans widely used remote-sensing imagery classes and a non-image dataset of country characteristics.The paper organizes data by experiment, with fuller descriptions in Appendix A.6.
- Land cover classification: Land-cover classification combines 0.6 m four-band NAIP aerial imagery from California with 30 m CDL ground-truth labels spanning 132 classes.The NAIP region contains 66 observed CDL classes, and CDL labels are upsampled to NAIP resolution.
- Latent space interpolation and visual analogy: Visual analogy evaluation uses median Landsat 8 composites with 7 spectral bands across urban and rural areas of San Francisco, New York City, and Boston.Landsat 8 provides moderate-resolution 30 m multispectral imagery on a 16-day collection cycle.
- Land cover classification: Table 1 compares Tile2Vec features with unsupervised baselines using RF, LR, and MLP classifiers trained with n = 1000 or n = 10000 labels.Mean accuracies and standard deviations are reported over 10 trials.
- Poverty prediction in Uganda: Poverty prediction uses median Landsat 7 composites from Uganda collected during 2009–2011, with five spectral bands and labels from the 2011–12 LSMS survey.LSMS measures annual household- and village-level consumption expenditure used to determine international extreme-poverty standards.
- Worldwide country health index prediction: Country-characteristic prediction uses 73 real-valued features from the 2015 CIA World Factbook for 242 countries.The features include indicators such as infant mortality rate, GDP per capita, and crude oil production.
4 Experiments
Tile2Vec is evaluated across land-cover classification, poverty prediction, latent-space structure, and robustness experiments. Its learned features outperform competing unsupervised representations and, on classification, supervised CNNs trained with up to 50k labels.
- Land cover classification: Tile2Vec features outperform other unsupervised features with random forest, logistic regression, and multilayer perceptron classifiers using 1,000 or 10,000 labels.The comparison includes autoencoder, pretrained ResNet-18, PCA/ICA, and k-means features.
- Supervised learning comparisons: Logistic regression on Tile2Vec features beats supervised CNN architectures trained on as many as 50k CDL labels.The Tile2Vec and supervised ResNet CNNs share the same architecture, making the classification-layer comparison direct.
- Latent space interpolation: Tile2Vec embeddings produce increasingly developed tiles along linear interpolations between field and urban representations, using five nearest neighbors at each step.The interpolation experiment evaluates whether the latent space changes semantically smoothly.
- Training details: Tile2Vec training is stable, relatively insensitive to margins from 0.1 to 100, and resistant to overfitting with explicit regularization and data augmentation.The training loss consistently decreases and serves as a proxy for downstream feature quality.
- Visual analogies: Tile2Vec representations support visual analogies across San Francisco, New York City, and Boston through addition and subtraction in latent space.Starting from a rural NYC embedding, the reported arithmetic recovers urban NYC tiles by adding urban SF and subtracting rural SF.
- Poverty prediction: Averaged r2 was 0.496 ± 0.014 for Uganda consumption-expenditure prediction, versus r2 = 0.41 for a transfer-learning approach.The Tile2Vec result used publicly available 30 m daytime imagery, compared with proprietary 2.4 m imagery for the prior approach.
5 Related Work
Tile2Vec adapts ideas from unsupervised visual representation learning and distributional word representations to spatially distributed data. It treats geospatial neighborhoods as context and uses spatially sampled tile triplets rather than semantic labels.
- Distributional representations: Tile2Vec is inspired by Word2vec and GloVe, treating multispectral image tiles as words and geographic neighborhoods as their contextual company.The approach transfers the distributional hypothesis from natural language to remote sensing data.
- Related visual methods: Unlike supervised patch2vec, Tile2Vec samples spatial triplets from remote sensing data without relying on annotated semantic-segmentation labels.The related patch2vec method learns globally consistent patch representations for natural images using supervision.
- Unsupervised visual learning: The broader unsupervised visual-learning landscape includes variational autoencoders, generative adversarial networks, autoregressive models, and representation-learning approaches.These model families aim to learn structure or data distributions from training samples without the paper’s specific spatial-triplet formulation.
6 Conclusion
The paper presents Tile2Vec as an unsupervised feature-learning method for spatially distributed data across imagery and non-image tasks. Its conclusion emphasizes broad empirical performance while identifying temporal information as an important direction for future work.
- Conclusion: Tile2Vec is evaluated from land-cover classification through poverty prediction and applies to aerial, satellite, multispectral, RGB, and non-image datasets.The conclusion describes the method as spanning moderate to high resolution and varied data modalities.
- Conclusion: Tile2Vec outperforms other unsupervised feature-extraction techniques and supervised CNNs trained on 50k labeled examples in a difficult classification task.This is the paper’s highlighted classification result.
- Future work: The paper focuses on spatial coherence, while temporal patterns such as seasonality and crop cycles remain planned future work.Many geospatial datasets contain sequences from the same locations collected over time.
A.1.1 Autoencoder
The convolutional autoencoder learns reconstructions of NAIP image tiles, including examples from a hold-out validation set unseen during training.
- The autoencoder uses three convolutional and two linear encoder layers, plus one linear and three deconvolutional decoder layers.Training uses batch size 100.
- The autoencoder is able to learn good reconstructions even for unseen hold-out examples.
- Hold-out NAIP tiles are reconstructed after training without being seen during autoencoder training.The examples compare original tiles in the top rows with reconstructions in the bottom rows.
A.1.2 DCGAN
The DCGAN generates visually reasonable NAIP samples, but discriminator features provide weak classification performance.
- The DCGAN is trained on 300k 50 × 50 × 4 NAIP image tiles.
- The generated samples are visually reasonable when their RGB channels are visualized.
- Classification accuracy using features learned by the trained discriminator does not exceed 25%.
A.1.3 PCA, ICA, K-means
The paper compares conventional unsupervised features and Tile2Vec across classification, poverty prediction, country-characteristic prediction, and embedding-space analyses. Tile2Vec features support strong downstream performance and meaningful spatial-semantic structure, with neighborhood choice affecting results.
- PCA, ICA, K-means: Tile2Vec paired with classifiers performs better than state-of-the-art supervised architectures for labeled dataset sizes up to 50k.
- PCA, ICA, K-means: 0.496 ± 0.014 r2 is achieved for Uganda consumption-expenditure prediction, compared with a reported transfer-learning r2 of 0.41.
- PCA, ICA, K-means: Tile2Vec embeddings organize country and land-cover data using both spatial and characteristic similarities, including clustering tiles from the same CDL class.
- PCA, ICA, K-means: Embedding arithmetic retrieves urban New York tiles from a rural New York tile after adding and subtracting corresponding San Francisco embeddings.
- PCA, ICA, K-means: A 100-pixel neighborhood is optimal on the NAIP land-cover task, while sampling neighbors and distant tiles from anywhere yields majority-class-level accuracies.