Source-linked AI summary
Automatic Discovery and Geotagging of Objects from Street View Imagery
Vladimir A. Krylov, Eamonn Kenny, Rozenn Dahyot
TL;DR
Accurate object inventories and geolocations matter for applications such as navigation, urban planning, and asset monitoring, but the paper identifies a lack of solutions using street-view imagery alone. It combines segmentation, monocular depth estimation, and MRF-based triangulation to geotag recurring stationary objects. Experiments report geotagging accuracy within 2 meters, comparable to a single-frequency GPS receiver.
Problem
The paper addresses automatic discovery and geotagging of recurring objects using street-view images alone, for which the authors say no such solution had been proposed to their knowledge.
Source: §1 · p. 1Method
Results
Geotagging accuracy is within 2 meters, similar to that obtained with a single-frequency GPS receiver.
Source: §5 · p. 8Takeaways & Limitations
The triangulation-based MRF helps avoid duplicates and reduce false positives while handling multi-object scenes, and can be combined with other segmentation or depth pipelines.
Source: §5 · p. 8The method assumes objects are at least 1 m apart and is evaluated on traffic lights and telegraph poles.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Many applications such as autonomous navigation, urban planning and asset monitoring, rely on the availability of accurate information about objects and their geolocations. In this paper we propose to automatically detect and compute the GPS coordinates of recurring stationary objects of interest using street view imagery. Our processing pipeline relies on two fully convolutional neural networks: the first segments objects in the images while the second estimates their distance from the camera. To geolocate all the detected objects coherently we propose a novel custom Markov Random Field model to perform objects triangulation. The novelty of the resulting pipeline is the combined use of monocular depth estimation and triangulation to enable automatic mapping of complex scenes with multiple visually similar objects of interest. We validate experimentally the effectiveness of our approach on two object classes: traffic lights and telegraph poles. The experiments report high object recall rates and GPS accuracy within 2 meters, which is comparable with the precision of single-frequency GPS receivers.
1. Introduction
The paper targets automatic discovery and geotagging of recurring stationary roadside objects from street-view imagery alone. Its pipeline combines semantic segmentation, monocular depth estimation, and MRF-based triangulation to map objects, including visually similar instances.
1. Introduction
- The target is compact, stationary objects along roads that can be assigned a single geotag, including street furniture, facade elements, and minor landmarks.The paper identifies precise inventory and GPS mapping of these objects as a relevant task and notes that users are currently encouraged to contribute such data manually.Source: §1 · p. 1
- The pipeline uses fully convolutional networks for semantic segmentation and monocular depth estimation, then combines depth with geometric triangulation in a Markov Random Field.The geotagging model is presented as a novel component of the pipeline.Source: §1 · pp. 1–2
- The approach handles objects that may be partially or completely occluded without explicit object-shape modeling or object-location patterns, assuming objects are at least 1 m apart.Experiments evaluate traffic lights and telegraph poles.Source: §1 · p. 2
- The MRF-based procedure uses monocular depth estimates to resolve complex scenes containing multiple instances of identical objects, unlike previous works.The geotagging module is modular and can be paired with alternative segmentation and depth techniques or pretrained solutions for particular object families.Source: §1 · p. 2
2. Related work
Prior work uses street-view imagery for applications ranging from mapping and navigation to image referencing and visualization. Related methods also address object extraction, object mapping, and depth estimation, but face challenges with matching visually similar objects and with the demands of stereo imagery.
2. Related work
- Street-view imagery has been combined with aerial data for road segmentation and tree detection, used to geolocate query images, and paired with Twitter for visualization and 3D rendering.Other cited applications include mapping, navigation, and image referencing.Source: §2 · p. 2
- Methods for mapping manholes, telecom assets, and road signs use triangulation, but matching performs poorly when multiple identical objects appear together.SIFT matching can be unreliable for small objects under strong viewpoint changes.Source: §2 · p. 2
- Fully convolutional networks support dense segmentation by retaining location information through convolutional layers and can produce bounding boxes or segmentation maps [18].The review traces segmentation approaches from pattern analysis and classifiers to machine-learning methods including CNNs [9] [19].Source: §2 · p. 2
- Stereo approaches estimate depth from multiple views but rely on camera-position or trajectory assumptions and typically need rich imagery, while monocular FCNN methods infer approximate depth from color bands [6] [12].Structure-from-Motion is also cited as a scene-reconstruction approach.Source: §2 · pp. 2–3
3. Object discovery and geotagging
The pipeline segments street-view images, estimates camera-to-object distances, and combines these observations in an MRF to geotag recurring objects. Its multi-view formulation uses depth estimates to address scenes with multiple visually similar objects.
3.1. Object segmentation
- The segmentation FCNN produces pixel-level object masks, supports one or multiple object classes, and applies extra false-positive penalization because false-positive pixels can create false object discoveries.The described FCN-8s architecture combines coarse segmentation maps with finer appearance information during upsampling.Source: §3.1 · p. 3
3.2. Monocular depth estimation
- A monocular depth FCNN estimates dense depth maps for images containing detected objects, and each object's distance is the 10%-trimmed mean of its segment-pixel depths.Trimming is intended to reduce sensitivity to segmentation errors, especially along object borders.Source: §3.2 · p. 3
3.3.1 Location from single view
- For a single view, the method combines the segmented region's orientation with its estimated distance to calculate an object geotag, but estimates are inaccurate and repeated sightings create redundant, inconsistent locations.Objects less than 1 meter apart may be recovered as one instance with a unique geotag.Source: §3.3.1 · p. 3
3.3.2 Location from multiple views
- The custom MRF labels pairwise view-ray intersections and uses their irregular neighborhoods to geotag complex scenes where visually similar objects defeat reliable image matching.The method combines approximate camera-to-object distances with triangulation; interpreting the intersections matters because grouping them without interpretation can produce wrong object maps.Source: §3.3.2 · p. 4, Figure 3 · p. 4
- Its energy penalizes disagreement with depth estimates, likely occlusion or excessive spread among intersections, and rays without positive intersections.The formulation treats distant positive intersections on one ray as unlikely for compact objects and penalizes them accordingly.Source: §3.3.2 · pp. 4–5
- The MRF is optimized from an empty configuration with Iterative Conditional Modes, then positive intersections within 1 meter are clustered and averaged into final geotags.The clustering distance corresponds to the stated object-sparsity assumption.Source: §3.3.2 · p. 5
4. Experimental study
The study tests automatic detection and geotagging of traffic lights and telegraph poles in Google Street View imagery, using panoramas queried along road-center grids at 5 m steps.
4.1. Traffic lights
- Within 25 m of the camera, traffic-light instance-level precision is .951 and recall is .981; objects farther away are conservatively ignored because image-processing reliability drops.The reported instance recovery criterion is that over 25% of an object's pixels are labeled correctly.Source: §4.1 · p. 6
- Both traffic-light object recovery precision and recall exceed .92, outperforming the less-than-.85 results reported for telecom asset recognition.Geotagging produced 70 objects in 51 clusters from 179 single-view instances.Source: Table 1 · p. 6, §4.1 · p. 7
4.2. Telegraph poles
- Telegraph-pole segmentation uses a custom two-stage training process: broad tall-pole training followed by fine-tuning on 500 pixel-level annotated telegraph-pole images.The training also adds 15K GSV scenes without telegraph poles but with lampposts and electricity poles.Source: §4.2 · p. 7
- Within 25 m of the camera, telegraph-pole object-level test recall is .979 and precision is .927.The distance restriction supports distinguishing telegraph poles from other tall poles whose subtle differences are not visible farther away.Source: §4.2 · p. 7
- The method achieves 95.8% average relative-distance precision (18% standard deviation), improving on 70% average reported in; its object-position error has a 2.07 m 95%-empirical confidence interval, comparable to 1.89 m for single-frequency GPS receivers.The accuracy statistics use reference data collected on site with a GPS receiver.Source: §4.2 · p. 8, Table 2 · p. 8
- Pole ground-truth GPS accuracy is nonhomogeneous, so these results validate precision and recall rather than GPS-tagging accuracy; visual-matching methods cannot map about 35% of pole-containing scenes.Multiple visually identical poles in a scene make those methods inefficient.Source: §4.2 · p. 8
5. Conclusions
The pipeline combines segmentation, depth estimation, and triangulation-based MRF geotagging, with high object recall and geotagging accuracy within 2 meters. The authors identify camera-position noise as the main accuracy constraint and propose unified segmentation and depth training as future work.
5. Conclusions
- The pipeline combines existing deep-learning pipelines for segmentation and depth with a novel triangulation-based MRF that handles multiobject scenes and helps avoid duplicates and false positives.The geotagging module can be combined with other segmentation or depth-estimation pipelines, and experiments demonstrate high object recall.Source: §5 · p. 8
- Geotagging accuracy is within 2 meters, similar to a single-frequency GPS receiver; camera-position noise is the main source of GPS inaccuracy.The authors suggest suppressing camera-position noise or outliers to improve mapping accuracy.Source: §5 · p. 8
- A unified FCNN for segmentation and depth is proposed as future work and may partially or fully relax sparsity assumptions by improving input to triangulation-based geotagging.Source: §5 · p. 8
Supplementary material
Horizontal stitching artifacts can disconnect an object's upper part, making its geo-orientation unreliable; the segmentation output is masked above the top stitching line to avoid incorrect geotagging.
Supplementary material
- Masking detections above the top stitching line avoids incorrect geotagging when stitching artifacts disconnect an object's upper part and prevent reliable geo-orientation inference.Source: Panoramic stitching · p. 10
2-stage telegraph pole segmentation training
Telegraph-pole segmentation is challenging because no public dataset supports sufficient training to distinguish these poles from similar vertical objects. A two-stage training procedure addresses this challenge, and the video reports successful discovery of 37 of 38 poles with segmentation false positives removed by geotagging.
2-stage telegraph pole segmentation training
- The absence of a sufficiently large public dataset makes it difficult to distinguish telegraph poles from electricity poles, lampposts, and sparsely vegetated trees.The target poles are wooden, naturally textured, not used for lamps, and not shared with electricity distribution; they may be overgrown or surrounded by vegetation.Source: 2-stage telegraph pole segmentation… · p. 10
- Training first detects all poles using proprietary and public datasets, then uses 500 custom pixel-level segmentations and negative examples to train the final telegraph-pole segmenter.The first stage is needed because the 500-image custom dataset alone is insufficient to distinguish poles from strong vertical features such as facades, fences, and trees.Source: 2-stage telegraph pole segmentation… · p. 10
- Examples show correct segmentation in complex scenes, while electricity poles, lampposts, and trees are rejected as non-telegraph poles.Source: Figure 8 · p. 10, 2-stage telegraph pole segmentation… · p. 10
Video
- 37 of 38 telegraph poles along the analyzed road stretch are discovered, and geotagging rejects all segmentation false positives.Detection and geotagging use full 360-degree panoramas, although the video visualization shows the frontal 90-degree field of view.Source: Video · p. 10