Source-linked AI summary
MESSY STREETS: A Benchmark for Geocoding Real-World Addresses
Edward Gaere, Florian von Wangenheim
TL;DR
Geocoders remain insufficiently characterized on authentic, non-canonical addresses, which are common in real-world data. MESSY STREETS builds a Web Data Commons benchmark with independently established reference locations and measures candidate retrieval, localisation, and surface-form divergence. Commercial systems mainly outperform open-source systems through higher candidate return on noisy addresses, while positional accuracy is broadly comparable once candidates are returned.
Problem
Robustness to authentic non-canonical addresses is poorly characterized because existing evaluations mainly use clean or synthetically perturbed data.
Method
MESSY STREETS evaluates geocoders on verbatim Web Data Commons addresses with reference locations from OpenAddresses or OpenStreetMap, separating candidate return from localisation accuracy.
Results
Commercial providers almost always return results whereas open-source geocoders often do not; localisation-accuracy differences are comparatively small, and surface divergence especially harms open-source systems.
Takeaways & Limitations
Address normalisation and preprocessing could substantially narrow the gap between open-source and commercial geocoders on messy inputs.
Takeaways & Limitations
The benchmark is geographically concentrated in North America and Europe and dominated by Latin-script addresses.
Abstract
from arXiv · showhide
We introduce MESSY STREETS, a benchmark for evaluating geocoders on verbatim web addresses, with existence verification and controlled measurement of surface-form divergence. Unlike conventional benchmarks based on clean or synthetically perturbed addresses, MESSY STREETS contains addresses whose surface forms diverge from canonical representations and whose components may be missing, repeated, malformed, or incomplete. The benchmark is constructed from the December 2024 Web Data Commons corpus, with reference locations established from OpenAddresses or OpenStreetMap. The strongest commercial geocoders outperform open-source systems by up to 49 percentage points in recall. This gap is driven primarily by differences in candidate return rates on non-canonical addresses; once a candidate is returned, positional accuracy is broadly comparable across systems. Non-canonical surface form alone accounts for up to 25 percentage points of recall loss. Examining Nominatim's query-processing pipeline, we show that its conjunctive matching lets a single unrecognised token zero an otherwise valid query. The results demonstrate that geocoder choice is a consequential design decision for applications processing noisy address data, and that normalisation and preprocessing could substantially narrow the gap between open-source and commercial geocoders.
1 Introduction
MESSY STREETS addresses the limited evidence on geocoder robustness to authentic, non-canonical web addresses. It introduces a reproducible benchmark and measures how surface-form divergence affects geocoding performance.
- Real-world addresses are often incomplete, abbreviated, misspelled, reordered, transliterated, or contaminated by stray characters and encoding artefacts.Such addresses arise across CRM systems, directories, spreadsheets, web forms, and large-scale web data.
- Existing evaluations provide limited evidence because they predominantly use clean datasets or synthetic perturbations rather than naturally occurring addresses.
- MESSY STREETS introduces a reproducible benchmark of verbatim web addresses derived from Web Data Commons and validated against open reference data.
- The benchmark evaluates twelve commercial and open-source geocoders across multiple spatial precision levels and quantifies lexical divergence from canonical address forms.
- Non-canonical surface forms alone account for up to a 25-percentage-point reduction in recall.
2 Related Work
Prior geocoder evaluation and address-parsing research largely centers on clean or curated addresses. MESSY STREETS instead targets messy real-world inputs and spatial geocoding performance.
- No standard benchmark exists for evaluating geocoding on messy, real-world data.
- Geocoder evaluation: Existing geocoder evaluations compare services on clean reference addresses or test robustness using synthetically injected typos and component errors.
- Address parsing: Address-parsing systems label components such as house number, street, city, and country rather than directly measuring spatial resolution.
- Parsing and standardising addresses toward canonical form could ultimately improve geocoding on messy inputs.
3 Benchmark Construction
MESSY STREETS constructs three non-overlapping tiers of web-derived addresses that balance authenticity with independent existence and component verification. The benchmark preserves noisy surface forms while supporting street-level evaluation.
- Construction methodology: The benchmark starts from December 2024 Web Data Commons records, extracts streetAddress and additional address components, and discards records without a streetAddress.The street filter may lose addresses whose street name appears only in another component.
- Gold, Silver, and Raw tiers: MESSY STREETS releases three non-overlapping tiers of 10K records, sampled without stratification and sharing only the presence of a street address.This design balances authenticity and verifiability while retaining the underlying Web Data Commons distributions.
- Gold, Silver, and Raw tiers: The gold tier contains 10K verbatim addresses verified for existence, with key components individually checked for plausibility.
- Gold, Silver, and Raw tiers: Producing the 10K gold records required evaluating 28K candidates, with a 64% rejection rate; 61% matched OpenAddresses and 39% matched OpenStreetMap.
- Gold, Silver, and Raw tiers: The silver tier is existence-verified but omits component verification, while the raw tier is neither existence-verified nor component-verified and is excluded from reported experiments.
- Descriptives & Biases: Geographic coverage is concentrated in North America (49%) and Europe (43%), while Latin-script addresses dominate each tier at 98%, 96%, and 94%, respectively.
4 Experiments & Results
The experiments evaluate geocoders on messy addresses by separating candidate retrieval from localisation accuracy. Commercial systems generally retrieve candidates more reliably, while surface-form divergence is a major source of failure, especially for open-source systems.
- Evaluation setup: Candidate Return Rate measures whether a geocoder returns at least one candidate, providing an upper bound on recall before localisation accuracy is assessed.A query returning no candidate cannot be correct, whereas a returned candidate may still be mislocated.
- Evaluation setup: Twelve geocoders are evaluated on MESSY STREETS gold and silver tiers using verbatim address strings submitted through public free-text APIs.The evaluation includes nine commercial and three open-source geocoders, with repeated runs on shared address sets.
- Overall results: Commercial providers almost always return a result, whereas open-source geocoders often do not; localisation-accuracy differences are comparatively small.The results identify retrieval, rather than coordinate placement, as the primary challenge.
- Surface-form robustness: Surface-form divergence consistently degrades performance and has an especially pronounced effect on open-source geocoders.Table 5 compares candidate return and GH6 accuracy across component verification and divergence from canonical forms.
- Failure analysis: Nominatim failures arise because conjunctive matching requires all input tokens to match simultaneously, so one unrecognised token can zero an otherwise valid query.Problematic tokens include unfamiliar names, locale conventions, suite designators, sublocalities, route numbers, duplicates, and verbose administrative components.
5 Conclusion
MESSY STREETS identifies robustness to non-canonical address forms as a major differentiator between geocoders. It concludes that preprocessing and normalisation are promising directions for narrowing the performance gap.
- Conclusion: Robustness to non-canonical address forms is a major differentiator between geocoders.Surface-form divergence has a greater impact on performance than component verification.
- Conclusion: Address normalisation and preprocessing could substantially narrow the gap between open-source and commercial geocoders.Suggested techniques include sub-unit stripping, exonym normalisation, and administrative-component deduplication before engine submission.
- Conclusion: Future work should test whether similar conjunctive-matching mechanisms explain recall gaps in Pelias and Photon.The conclusion also calls for rigorous testing of pre-parsing techniques before engine submission.