Source-linked AI summary
A Generalizable and Accessible Approach to Machine Learning with Global Satellite Imagery
Esther Rolf, Jonathan Proctor, Tamma Carleton, Ian Bolliger, Vaishaal Shankar, Miyabi Ishihara, Benjamin Recht, Solomon Hsiang
TL;DR
SIML could help estimate socioeconomic and environmental conditions globally, but resource requirements limit access. The paper develops MOSAIKS, which shares a single unsupervised satellite-image encoding across tasks and lets users fit linear regressions to their own labels. It reports competitive performance with deep learning at much lower computational cost, alongside global scaling, label super-resolution, and uncertainty analysis.
Problem
Resource-intensive SIML pipelines limit access to satellite-based estimates, particularly in low-income settings with limited technical capacity.
Method
MOSAIKS centrally computes task-agnostic random convolutional features once, then users fit task-specific linear regressions using their own ground-truth labels.
Results
A single feature set generalizes across diverse prediction tasks with performance competitive with deep-learning models, while requiring orders of magnitude less computation for new tasks.
Takeaways & Limitations
Shared features and simple regression can broaden access to global SIML, support stress-testing, and provide label super-resolution and multi-sensor extensions.
Takeaways & Limitations
Custom-designed SIML pipelines and ground-based surveys remain important in some contexts, with MOSAIKS serving as a complement rather than a universal replacement.
Abstract
from arXiv · showhide
Combining satellite imagery with machine learning (SIML) has the potential to address global challenges by remotely estimating socioeconomic and environmental conditions in data-poor regions, yet the resource requirements of SIML limit its accessibility and use. We show that a single encoding of satellite imagery can generalize across diverse prediction tasks (e.g. forest cover, house price, road length). Our method achieves accuracy competitive with deep neural networks at orders of magnitude lower computational cost, scales globally, delivers label super-resolution predictions, and facilitates characterizations of uncertainty. Since image encodings are shared across tasks, they can be centrally computed and distributed to unlimited researchers, who need only fit a linear regression to their own ground truth data in order to achieve state-of-the-art SIML performance.
1 Introduction
SIML can transform satellite imagery into large-scale estimates, but its resource requirements restrict accessibility, especially where technical capacity is limited. The paper introduces a shared, task-agnostic image encoding that lets non-experts use linear regression for diverse tasks at competitive performance.
- SIML supports estimates of forest cover, land use, poverty rates, and population densities from satellite imagery.
- Resource-intensive requirements for domain expertise, remote sensing, engineering, imagery access, and sophisticated model development limit SIML accessibility and use.
- A one-time task-agnostic encoding converts each satellite image into features that users can pair with ground-truth labels using linear regression on a personal computer.
- The same feature set performs competitively with deep-learning methods across diverse prediction tasks and scales globally.
- Centralized feature computation lets researchers share imagery-derived features while avoiding the costs of obtaining, storing, manipulating, and processing imagery themselves.The approach separates unsupervised feature construction from task-specific model fitting.
2 Multi-task Observation using Satellite Imagery & Kitchen Sinks
MOSAIKS separates satellite-image featurization from task-specific regression, allowing centrally computed representations to support many users and prediction tasks. Random convolutional features encode imagery once, after which users fit linear models to their own labels without manipulating images.
- MOSAIKS targets ground-condition prediction from satellite imagery using limited task-specific ground truth and basic computational resources.
- A fixed featurization step maps images to vectors, while a regression step learns task-specific coefficients mapping features to outcomes.
- One unsupervised featurization can be centrally executed and reused across many tasks through repeated regression by independent users.
- Users query stored feature vectors, merge them with independently collected labels, fit linear regression coefficients, and generate predictions from the full feature table.
- Random convolutional features apply random kitchen sinks to satellite imagery, producing a shared mathematical summary that works across prediction tasks without task-specific modification.
- MOSAIKS avoids deep-neural-network training and label requirements for unsupervised encoding while gaining computational advantages through shared computation and storage.
3 Results
MOSAIKS uses one task-independent satellite-image encoding to support diverse predictions, matching deep-learning approaches at much lower training cost while extending to global, spatially extrapolative, super-resolution, and uncertainty-aware applications.
- Generalization and computational efficiency: 250 to 10,000× faster to train, MOSAIKS remains competitively accurate with a ResNet-18 CNN across all seven US tasks.The speed advantage depends on whether ridge regression runs on a laptop or the same cloud node used for CNN training.
- Generalization and computational efficiency: A single set of task-independent features supports predictions across diverse US tasks, with accuracy and speed favorably comparing to existing SIML approaches.The experiments cover generalization across tasks, computational comparisons, limited data, spatial extrapolation, global prediction, and survey reconstruction.
- Resource sensitivity: K = 200 recovers 55% to 89% of the observable signal obtained with K = 8,192, while increasing K above 1,000 provides only minor predictive gains.The retained share ranges from 55% for income to 89% for nighttime lights.
- Spatial generalization: MOSAIKS outperforms spatial interpolation in 5 of 7 tasks, while performance varies with the size of spatial extrapolation and task.Forest cover, population density, and nighttime lights decline minimally with distance; elevation declines steadily, while income, road length, and housing price decline moderately before stabilizing.
- Global and applied predictions: At global scale, MOSAIKS predicts forest cover, elevation, population density, and nighttime lights with R2 values of 0.85, 0.45, 0.62, and 0.49, respectively.The global experiment uses K = 2,048 label-independent features and 338,781 training-validation locations plus 84,692 test locations.
- Global and applied predictions: Across 12 American Community Survey tasks, average R2 is 0.34, with values from 0.06 to 0.52, while some variables may remain recoverable only through ground surveys.The authors contrast these low-cost predictions with an ACS deployment costing over $200 million annually.
4 Discussion
MOSAIKS provides practical generalization across tasks while remaining faster and simpler than many deep-learning alternatives. Its interpretable design supports global-scale prediction, finer-resolution estimates, uncertainty stress tests, and broader data integration.
- MOSAIKS achieves practical generalization across tasks with performance competitive with deep learning models optimized for a single task.
- Orders of magnitude less computation time and ∼6-500 times compression support planet-scale analyses and wider access to SIML.The authors connect these gains to democratizing global-scale SIML technology.
- MOSAIKS can complement custom-designed SIML pipelines and ground-based surveys, including by guiding slower systems or extending survey range and resolution.
- Low-cost, high-speed retraining enables stress tests of sensitivity to model parameters, training-point counts, and spatial extrapolation.
- Analytically interpretable components support label super-resolution and integration of multiple sensors through feature concatenation.The paper conjectures that additional diverse data may improve tasks not fully resolved by daytime imagery alone.
- A publicly available global cross-section of features using 2019 Planet Labs imagery is presented as a step toward continuously encoded and shared global information.
Supplementary Materials Appendix
The supplementary materials document the evaluation, data, experiments, comparisons, and algorithmic foundations underlying MOSAIKS. They include dedicated sections on spatial generalizability, benchmarking, and computational costs.
- The supplementary materials summarize methods, data, experiments, and comparisons with other approaches in greater depth.
- They also describe the intuition and mechanics of MOSAIKS’s algorithms in greater detail.
- Section S.3.8 addresses generalizability across space and comparison to kernel-based interpolation.
- Sections S.4.1 and S.4.2 cover performance benchmarking and cost comparison, respectively.
S.1 Methods summary
The supplementary materials provide additional implementation details, experimental procedures, and theoretical foundations for MOSAIKS. Full methodological details are distributed throughout Section S.3.
- The appendix adds information about MOSAIKS’s implementation and experimental procedures.
- It describes the theoretical foundation underlying MOSAIKS.
- Full details on the methodology are provided throughout Section S.3.
Implementation of MOSAIKS
MOSAIKS converts satellite images into task-agnostic features using random convolutional patches, then distributes those features for linear regression. Its configuration trades storage and computation against predictive performance.
- MOSAIKS generates task-agnostic feature vectors by convolving randomly sampled M × M × S patches across each satellite image.
- Each convolution uses an inner product followed by a ReLU activation, and the resulting values are averaged across the image to form each feature.The main analysis uses M = 3 and S = 3 red, green, and blue bands.
- The resulting feature space has dimension K, with the main analyses using K = 8,192 features.
- A one-time unsupervised featurization can be centrally computed and distributed in tabular form, after which users link features to labels and fit linear regressions.
- K = 8,912 features yield roughly 6 to 1 compression, while storage and computational cost can be traded off against performance by changing the feature count.
Experimental procedures
The experiments evaluate MOSAIKS across diverse tasks, under varying spatial and data conditions, and at global scale, including label super-resolution.
- US imagery is aligned with labels on a ∼1km × 1km grid designed to prevent spatial overlap between observations.
- The US experiments use 20,000 hold-out test cells and 80,000 training and validation cells, with alternative spatial and population-weighted sampling schemes.
- Sensitivity analyses vary geographic train-test distance, feature dimension K, and training-set size N.
- The global experiment targets N = 1,000,000 locations but realizes N = 423,476 because imagery and label data are missing outside the US.
- Global predictions cover forest cover, elevation, population density, and nighttime lights using the same approach as the US analysis.
- Label super-resolution predictions are generated for forest cover and population density from un-pooled sub-image features, trained ridge weights, and Gaussian smoothing.
Theoretical foundations
MOSAIKS uses random kitchen sinks and convolutional features to construct a compact, task-independent representation of satellite imagery for linear prediction.
- Random kitchen sinks approximate arbitrary functions with finite features generated by nonlinear functions applied to inputs and random parameter draws.
- Fully expressive random features are inefficient for satellite images because each feature encodes more pixel interactions than empirically useful.
- MOSAIKS adapts random kitchen sinks by assuming most useful satellite-image information is represented in local image structure.
- Each feature uses a randomly selected image patch, convolves it across the image, applies ReLU, and averages the activation map.
- Convolutions of small patches are motivated by the tendency of objects of interest to occupy small image regions.
- A constant viewing perspective and frequent orthorectification support a simple translation-invariant, single-layer featurization scheme.
- Average pooling reflects that many studied labels can be approximately decomposed into sums of sub-image characteristics.
S.2 Data
The dataset design links satellite imagery to diverse ground-condition variables selected to test MOSAIKS across multiple outcome types.
- The tasks include averages of sub-image properties, variables inferred from visible objects, and underlying factors that determine visible materials.
- Each task uses an up-to-date and geographically complete public data source, although population data include sources dating as far back as 2005 for global coverage.
Labels
The labels span environmental, socioeconomic, infrastructure, and remotely sensed outcomes, with documented resolution, source, and data-quality differences.
- Global raster resolutions for forest cover, elevation, population density, and nighttime lights are stated at the equator and vary in Euclidean distance with latitude.
- Forest-cover labels measure vegetation taller than 5m at roughly 30m × 30m resolution and derive from Landsat-based growing-season imagery.
- Nighttime-lights labels are 2015 annual VIIRS composites filtered for stray light, lunar illumination, clouds, fires, boats, and other artifacts.
- Daytime imagery predicts nighttime luminosity from independent data sources, relying on associations between visible daytime objects and nighttime emissions.
- Road-length labels sum recorded road lengths in each grid cell, but recording accuracy varies regionally and often omits private roads and dirt trails.
- Housing-price labels require substantial cleaning because transaction and assessment data come from states and counties with widely varying regulations and standards.
- Public housing-price labels retain only grid cells with at least 30 qualifying sales, reducing the dataset from N = 80,420 to N = 52,355.
S.3 Methods
This section outlines the methods for defining samples, constructing labels and features, separating training and evaluation data, and testing predictive performance.
- The methods define samples, construct labels and features, separate training and evaluation data, and train and test models.
S.3.1 Grid definition and sampling strategy
The study uses a standardized global grid to link satellite imagery with labeled data, then samples observations either uniformly or according to population weights.
- A single global grid provides a standardized observation unit linking satellite imagery and labeled data across tasks.Grid cells are square in physical space, with angular extent varying across latitudes.
- US grid cells vary from 1.39 km at the southern edge to 0.98 km at the northern edge.
- 100,000 continental US observations are sampled using either uniform-at-random or population-weighted strategies.Uniform sampling suits tasks with broad geographic variation, while population weighting targets populated areas.
- Forest cover, elevation, and population density use uniform-at-random sampling, while nighttime lights, income, road length, and housing price use population-weighted sampling.
S.3.2 Assigning labeled data to sampled imagery
Labels are assigned by spatially overlaying raw task data with the imagery grid and applying aggregation rules suited to each data type.
- Raw labeled data are spatially overlaid with the custom imagery grid, requiring task-specific aggregation or disaggregation procedures.
- Raster tasks use mean values within each grid cell, whereas road labels use the total recorded road length.The raster tasks include forest cover, elevation, population density, and nighttime lights.
- Housing labels are average price per square foot across observed sales within each grid cell.
- Income labels are area-weighted averages of block-group incomes, with intersection-area weights normalized within each grid cell.
- Production-scale users may match arbitrary label formats and resolutions to the pre-computed feature grid, including sparse points matched to the nearest grid-cell centroid.
S.3.3 Featurization of satellite imagery
MOSAIKS converts satellite images into shared nonlinear feature vectors by convolving sampled patches, applying ReLU, and pooling activations; task-specific models then map features to outcomes.
- The pipeline samples random patches from training images, convolves them across each image, applies nonlinear activation, and pools activation maps.The process creates distributed image features before task-specific model fitting.
- Each activation-map value applies ReLU to the patch–image inner product plus a bias, and average pooling produces one scalar per patch–image pair.ReLU is defined as max{patch–image response, 0}.
- MOSAIKS represents each satellite image with a K-dimensional feature vector that can support task-specific outcome models.The features embed images in a shared feature space for multiple outcomes.
- The resulting linear relationship between labels and features can represent highly nonlinear image-to-label mappings because the features are nonlinear in the images.
- The main analysis uses 3×3×3 patches because M = 3 performs best across all seven tasks.Larger patches may capture larger spatial structure, but did not perform best in practice.
- MOSAIKS can be interpreted as a computationally tractable approximation to convolutional-kernel regression and as a shallow, wide CNN using random filters without backpropagation.
S.3.4 Data separation practices and cross-validation
MOSAIKS is evaluated through cross-validation, spatial separation, functional-form checks, feature and sample-size sensitivity, and comparisons with interpolation and heterogeneous models. These tests show stable transfer across folds and space, competitive performance, label super-resolution, and gains from modeling geographic heterogeneity, while leaving room for task-specific improvement.
- Cross-validation and task transfer: High R2 values across folds indicate that regression weights are strongly correlated within forest cover, elevation, and population-density tasks.Across tasks, weights are virtually uncorrelated, indicating that task-relevant features occupy orthogonal subspaces; experiments use K = 1,024 random features.
- Functional form: 10-64% R2 increases occur after logging highly skewed tasks, whereas road length, income, and elevation show 4-21% reductions in log form.The reported examples include population density, housing price per square foot, and nighttime lights among the highly skewed tasks.
- Feature and sample-size sensitivity: 100 features retain 81% to 96% of the R2 achieved with 8,192 features, while doubling features from 4,096 to 8,192 adds 0.010-0.026 R2.The smallest and largest gains are reported for forest cover and income, respectively; performance had not fully flattened by K = 8,192.
- Feature and sample-size sensitivity: 500 observations recover 56% to 87% of performance relative to N = 64,000 for road length and forest cover, while income and housing price require larger samples.Performance increases with training-set size but exhibits diminishing marginal returns.
- Spatial generalizability: Performance degrades only slightly as spatial separation between training and testing observations increases, suggesting image-label mappings transfer across regions.The checkerboard evaluation separates training and validation sets by increasingly large distances.
- Comparative evaluation: MOSAIKS outperforms kernel-based spatial interpolation and trained twelve predictive models in less than 45 minutes on a ten-core workstation.The interpolation comparison indicates that image spectral and structural content contributes beyond geographic location alone.
- Label super-resolution and heterogeneity: Over 50% of within-image label variation is explained at 2 × 2 super-resolution and over 30% at 32 × 32, despite training only on aggregate labels.The native forest-cover labels are approximately 30 m, just under 1/32 of the approximately 1 km image width.
- Label super-resolution and heterogeneity: Accounting for continental heterogeneity yields R2 values of 0.85, 0.45, 0.62, and 0.49 versus 0.80, 0.26, 0.48, and 0.41 for a pooled global model.The values correspond, in order, to forest cover, elevation, population density, and nighttime lights.