Source-linked AI summary
Improving Image Classification with Location Context
Kevin Tang, Manohar Paluri, Li Fei-Fei, Rob Fergus, Lubomir Bourdev
TL;DR
The paper addresses how to classify consumer images using location context when GPS coordinates are available in both training and testing. It develops five types of GPS-derived location features and incorporates them into a CNN, jointly learning visual features and feature interactions. Learning pooling radii for features that require pooling yields better performance within the CNN framework.
Problem
Method
It develops five types of GPS-derived location features and incorporates them into a CNN, jointly learning visual features and feature interactions.
Source: §1 · p. 2Results
Learning pooling radii for features that require pooling yields better performance within the CNN framework.
Source: §7 · p. 8Takeaways & Limitations
The publicly available YFCC100M-GEO100 annotations cover 88,986 images over 100 classes and enable model evaluation at scale.
Source: §1 · p. 2The experiments focus on images from the contiguous United States, although most features can be extended to the entire world.
Source: §3 · p. 2
Abstract
from arXiv · showhide
With the widespread availability of cellphones and cameras that have GPS capabilities, it is common for images being uploaded to the Internet today to have GPS coordinates associated with them. In addition to research that tries to predict GPS coordinates from visual features, this also opens up the door to problems that are conditioned on the availability of GPS coordinates. In this work, we tackle the problem of performing image classification with location context, in which we are given the GPS coordinates for images in both the train and test phases. We explore different ways of encoding and extracting features from the GPS coordinates, and show how to naturally incorporate these features into a Convolutional Neural Network (CNN), the current state-of-the-art for most image classification and recognition problems. We also show how it is possible to simultaneously learn the optimal pooling radii for a subset of our features within the CNN framework. To evaluate our model and to help promote research in this area, we identify a set of location-sensitive concepts and annotate a subset of the Yahoo Flickr Creative Commons 100M dataset that has GPS coordinates with these concepts, which we make publicly available. By leveraging location context, we are able to achieve almost a 7% gain in mean average precision.
1. Introduction
The paper tackles image classification using GPS context, motivated by cases where location helps distinguish visually ambiguous content. It develops location features and CNN architectures, and introduces a publicly available annotated dataset.
Motivation
- Location can make visually ambiguous images easier to classify, as geographic context identifies which example actually contains snow.The examples include Bonneville Salt Flats, Death Valley, Palo Alto, and New Hampshire.Source: §1 · p. 1
Location features
- The paper proposes five location-feature types that extend latitude and longitude and evaluates their effectiveness.Source: §1 · p. 2
CNN architecture
- The authors incorporate location features into a CNN to jointly learn visual features and interactions among feature types.They also learn parameters for constructing a subset of features within the same framework.Source: §1 · p. 2
Dataset
- YFCC100M-GEO100 provides publicly available annotations for location-sensitive concepts on a subset of YFCC100M, comprising 88,986 images over 100 classes.Source: §1 · p. 2
2. Related Work
Prior work spans image geolocation and the use of contextual or location-linked data in recognition. This paper differs by targeting a wider range of Internet concepts and evaluating general location features in a CNN.
Image geolocation
- Image-geolocation research includes static-camera and city-scale recognition, im2gps, place and landmark recognition, and graph-based approaches.Other work uses elevation maps and land-cover surveys, which the authors cite as inspiration for constructing features.Source: §2 · p. 2
Location and context for recognition
- Location-informed recognition has addressed small urban-object sets, bird categorization, and generic region recognition.Other context work examines tasks including organizing geotagged photos and recognizing city identity.Source: §2 · p. 2
Distinction from prior work
- Unlike those narrower recognition targets, this work addresses a wide range of Internet concepts with features not tied to one class or GIS source.It evaluates ways to incorporate the features into a CNN and introduces a large-scale geotagged dataset for training and evaluation.Source: §2 · p. 2
Visual context
- The paper uses GPS coordinates as context for image classification, complementing broader computer-vision work on contextual information.Source: §2 · p. 2
3. Our Approach
The task assumes GPS coordinates are available for both training and test images, and adds geographic context to image classification through CNN-integrated location features. The approach draws on geographic datasets and surveys as well as geotagged Internet data.
3.1. Neural network architecture
- Location features are concatenated into a CNN before the softmax layer, joining image and location information at a higher semantic level.The model takes an image and its longitude and latitude as input.Source: §3.1 · p. 2, Figure 2 · p. 3
3.2. GPS encoding feature
- GPS coordinates are encoded as an indicator vector for the grid cell containing each image's location.The grid cells are designed to be roughly square; the largest tested grid was 100x200, with cells of about 25x25 km.Source: §3.2 · p. 3
3.3. Geographic map feature
- Ten types of geographic maps supply normalized pixel colors from a 17x17 patch around each GPS coordinate, forming an 8670-dimensional feature.The map types include precipitation and elevation.Source: §3.3 · p. 3
3.4. ACS feature
- The ACS feature uses zip-code statistics pooled over five years to form a 21,038-dimensional vector for each location.The statistics cover demographics and other topics, including income, education, and living conditions.Source: §3.4 · p. 4
3.5. Hashtag context feature
- Hashtag context features count nearby geotagged Instagram images for each hashtag across multiple radii, then concatenate two types of normalized counts.One normalization captures relative hashtag frequency in an area and adjusts for photo density; the other relates a hashtag's local frequency to its frequency across the United States.Source: §3.5 · p. 4
3.6. Visual context feature
- Visual context features pool concept probabilities from nearby geotagged images across radii and concatenate two types of normalized values.A CNN estimates probabilities for 594 common Internet concepts, such as clothes, girl, and coffee.Source: §3.6 · p. 4
4. Learning the Optimal Pooling Radius
The radius learning layer addresses the possibility that not all pooling radii are informative. It learns a radius for each hashtag or concept from histogram features within the CNN.
Radius learning layer
- The radius learning layer automatically learns pooling radii, potentially reducing overfitting by focusing on informative radii and allowing the learned radii to be visualized.Different radii may be informative for rare versus common hashtags or concepts.Source: §4 · p. 5
- For each hashtag or concept, the layer uses a radius parameter to select an input value from a piecewise linear fit to the histogram feature.The fitted values are computed over the radii used for the original features.Source: §4 · p. 5
- Backpropagation combines the network's error derivative with the histogram function's local slope; the piecewise linear fit makes that slope easy to compute.The authors replicate the radius learning layer for each histogram function to represent multiple informative radii and weightings.Source: §4 · p. 5
5. Dataset
The dataset uses location-sensitive classes selected from Instagram hashtag distributions and manually verified, geotagged YFCC100M images. The resulting YFCC100M-GEO100 set contains 88,986 images across 100 classes.
5. Dataset
- The YFCC100M source has about 49 million geotagged images, but noisy Flickr tags require manual verification and the study uses images from the contiguous United States.The source dataset contains 100 million Creative Commons images.Source: §5 · p. 5
- The authors select the 100 classes with the greatest KL divergence between their Instagram hashtag distributions and the distribution of all images.The divergence measures distributional difference and is used to identify classes likely to have location-sensitive properties.Source: §5 · p. 5
- Instagram hashtag maps show beach tags near coasts and mountain tags outlining the Appalachians, but contain substantial noise.Source: Figure 5 · p. 6
- YFCC100M-GEO100 contains 88,986 annotated images across 100 classes, with at least 100 images per class, and is made publicly available.The classes range from objects to places and scenes, including autumn, beach, and whale.Source: §5 · pp. 5–6
6. Results
Location encodings improve on image-only baselines, and the strongest reported architecture combines pre-cat feature layers with learned pooling radii. Bayesian priors and added network depth can hurt overall performance through class-specific effects or overfitting.
6.1. Baseline methods
- GPS coordinates alone yield no significant gain, while GPS encodings gain around 2% and the ACS feature nearly 4% across all performance measures.Concatenating location features with baseline image features provides gains, indicating complementary information.Source: §6.1 · pp. 6–7, Table 1 · p. 6
- SVMs perform worse than the softmax classifier; chi-square kernels improve on linear kernels for histogram features but still do not exceed softmax.Source: §6.1 · p. 7
- Location priors do not improve overall results: most classes are hurt, although Disneyland gains more than 45% mean AP with either prior.The tested Flickr prior uses GPS-space k-nearest neighbors, while the Instagram prior uses a normalized hashtag-context histogram; the class prior is uniform.Source: §6.1 · p. 7
6.2. Architectures for feature combination
- A 256-dimensional pre-cat layer obtains almost a 6% gain across all performance measures, while additional pre-cat depth does not help significantly and increases parameters.The authors attribute the benefit of pre-cat layers to capturing within-feature relationships and reducing dimensionality.Source: §6.2 · p. 7, Table 2 · p. 7
- A 4096-dimensional post-cat layer slightly increases mean AP but decreases both normalized accuracy rates, and greater post-cat depth also reduces performance.The authors attribute the decreases to overfitting.Source: §6.2 · p. 7, Table 2 · p. 7
6.3. Learning the optimal pooling radius
- Learning pooling radii adds 0.5% mean AP for the best prior models; the authors select 256/- RL10 as their best model because adding a post-cat layer slightly overfits.Table 3 reports results for learning pooling radii; RL10 uses 10 replicas of the radius-learning layer.Source: Table 3 · p. 7, §6.3 · p. 8
- Location-specific classes such as Disneyland, casino, and Alcatraz improve substantially over the image-only baseline, while car brands are among the worst-performing classes.The authors suggest fine-grained car classes may be less location-specific or poorly handled by their features and model.Source: §6.3 · p. 8, Figure 8 · p. 8
- For most concepts, 10 learned radius replicas converge to three or fewer distinct radii, while building sometimes uses nearly all different radii.The authors suggest smaller radii may matter in urban areas and larger radii in rural areas for buildings.Source: §6.3 · p. 8, Figure 9 · p. 8
7. Conclusion
The conclusion presents location-context image classification through five GPS-derived features incorporated into a CNN, with learned pooling radii for features that pool over spatial neighborhoods. It also introduces a publicly available, manually annotated geotagged-image dataset and identifies time and social relationships as possible future context.
7. Conclusion
- The paper introduces image classification with location context using five features incorporated into a CNN.Source: §7 · p. 8
- For pooling-based features, the CNN can learn optimal radii, improving performance and providing insight into network parameters.Source: §7 · pp. 8–9
- The authors make YFCC100M-GEO100, a manually annotated dataset of geotagged images, publicly available.Source: §7 · p. 9
- Future work could use other image-associated information, including capture time and date or social relationships between users.Source: §7 · p. 9