Source-linked AI summary
Effective Multi-Query Expansions: Collaborative Deep Networks for Robust Landmark Retrieval
Yang Wang, Xuemin Lin, Lin Wu, Wenjie Zhang
TL;DR
Landmark retrieval can fail when a query photo depicts a landmark with low-quality geometry. The paper expands the query using latent topics and collaborative user-photo information, then learns deep features for the expanded set and database photos; experiments report state-of-the-art performance on real-world social landmark datasets.
Problem
Existing landmark retrieval methods assume query photos provide high-quality landmark shapes, but viewpoint and photography can produce low-quality geometry.
Method
The method constructs a multi-query set from latent-topic and user-community information, factorizes the collaborative user-photo matrix, and trains a deep network over the resulting pseudo-classes.
Results
The proposed collaborative deep network achieves state-of-the-art performance against low-level and mid-level landmark-retrieval methods on real-world social landmark photo datasets with user information.
Takeaways & Limitations
High-level features learned from collaborative latent factors provide the paper’s retrieval representation for both expanded queries and other landmark photos.
Takeaways & Limitations
The evaluation uses photo sets where biased landmark photos constitute a relatively small portion of each set, supporting validation under that data condition.
Abstract
from arXiv · showhide
Given a query photo issued by a user (q-user), the landmark retrieval is to return a set of photos with their landmarks similar to those of the query, while the existing studies on the landmark retrieval focus on exploiting geometries of landmarks for similarity matches between candidate photos and a query photo. We observe that the same landmarks provided by different users over social media community may convey different geometry information depending on the viewpoints and/or angles, and may subsequently yield very different results. In fact, dealing with the landmarks with \illshapes caused by the photography of q-users is often nontrivial and has seldom been studied. In this paper we propose a novel framework, namely multi-query expansions, to retrieve semantically robust landmarks by two steps. Firstly, we identify the top-$k$ photos regarding the latent topics of a query landmark to construct multi-query set so as to remedy its possible \illshape. For this purpose, we significantly extend the techniques of Latent Dirichlet Allocation. Then, motivated by the typical \emph{collaborative filtering} methods, we propose to learn a \emph{collaborative} deep networks based semantically, nonlinear and high-level features over the latent factor for landmark photo as the training set, which is formed by matrix factorization over \emph{collaborative} user-photo matrix regarding the multi-query set. The learned deep network is further applied to generate the features for all the other photos, meanwhile resulting into a compact multi-query set within such space. Extensive experiments are conducted on real-world social media data with both landmark photos together with their user information to show the superior performance over the existing methods.
1 Introduction
The paper addresses landmark retrieval when a query photo provides low-quality geometry. It expands the query with socially and topically related photos, then learns high-level collaborative deep features for robust retrieval.
- Problem: Existing landmark retrieval methods rely on landmark geometry and assume the query photo has a high-quality shape.Low-quality shapes can result from viewpoint, photography, or personal preference.
- Framework: The proposed framework discovers query topics and user communities, selects multiple landmark photos, and learns collaborative deep features for retrieval.Its three phases are topic and community discovery, multi-query construction, and deep feature learning with ranking.
- Framework: LDA-based topic discovery and user information support multi-query expansion to complement a biased landmark query.The expansion uses related users and photos to represent the same landmark from different perspectives.
- Representation Learning: A compact multi-query set is selected by ranking expanded photos in the learned high-level feature space against the query landmark.The selected representations are combined to compute final similarity scores over database photos.
- Representation Learning: The method trains a deep convolutional network over pseudo-classes formed from latent factors of a collaborative user-photo matrix.The resulting high-level features represent both the expanded query set and other landmark photos.
2 Related Work
Prior work uses low-level, mid-level, manually selected, or visually retrieved representations for landmark and query expansion tasks. This paper instead mines latent topics and user communities and learns high-level deep features over collaborative latent factors.
- Landmark Retrieval: Landmark retrieval research has used low-level feature matching, patch-level regions, mid-level attributes, and other heterogeneous feature spaces.These approaches differ in representation but target similarity-based landmark retrieval.
- Query Expansion: Earlier query expansion methods manually select multiple queries or retrieve top-k similar items, limiting their direct applicability to social media networks.The paper identifies social-media user communities and latent query topics as an alternative source of helpful queries.
- Query Expansion: Wang et al. used multi-query expansion with a mid-level pattern representation, whereas this work learns high-level deep features over the expanded set.The paper presents the deep representation as a non-trivial extension of that prior approach.
- Geo-tagging: The paper distinguishes its landmark-retrieval task from geo-tagging research, although both may explore user communities.Geo-tagging focuses on attaching geographical metadata to multimedia data and location-based services.
- Deep Feature Learning: The proposed method uses user information to form landmark-photo pseudo-classes from latent factors obtained by matrix factorization.These latent factors are intended to characterize user-related information beyond the original user-landmark matrix.
3 The Proposed Technique
The proposed technique expands a landmark query using latent topics and user-photo collaboration, then learns high-level deep representations to retrieve robustly across visual variation.
- Latent topic discovery and expansion: The first stage discovers latent topics in landmark photos with LDA and uses topic-related user groups to construct a multi-query expansion.Photos are modeled as observations in user albums, with each album represented as a mixture of latent topics.
- Latent topic discovery and expansion: LDA treats each album as a topic distribution and each topic as a distribution over photos, enabling photos sharing latent topics to be grouped.The topic assignments and distributions are latent, while the observed variables are the photos.
- Latent topic discovery and expansion: Candidate topics are selected using P(q|z) ≥ λ, with λ balancing topic inclusiveness against the inclusion of irrelevant topics.A small λ yields many potentially non-relevant topics, whereas a large λ may exclude true topics.
- Collaborative multi-query selection: Non-negative matrix factorization maps users and photos into a shared low-dimensional latent space, where inner products score photos for query-user expansion.The top-K photos by confidence score form a robust multi-query set describing the landmark from different aspects.
- Deep representation learning: The second stage fine-tunes CNNs on pseudo classes derived from collaborative latent factors to learn high-level discriminative landmark features.The learned network produces features for both the multi-query set and the remaining landmark photos.
- Deep representation learning: Multiple deep feature vectors from the expanded query are aggregated into an overall representation, with Fisher vector encoding used to support robust similarity search.The resulting representation is compared with database landmark photos using Euclidean distance.
4 Experiments
The experiments use Flickr and Picasa landmark-photo datasets containing user information, and examine retrieval under varied landmark viewpoints and capturing conditions.
- The experiments compare the proposed approach with varied baselines and state-of-the-art methods.
- Two datasets are built from Flickr and Picasa Web Album because they pair landmark photos with user information.
- The Flickr dataset covers landmark photos from 11 cities collected through city and landmark text-tag queries.
- The setup does not assume different users select the same landmark photo, because redundant multi-query expansions are considered uninformative.
- The data instead reflects the common case of one landmark photographed under different angles and capturing conditions.
4.2 Training and validation
Training and validation evaluate learned representations through retrieval metrics and comparisons with query-expansion, hypergraph, deep-learning, and handcrafted-feature baselines.
- Precision-recall is evaluated over top-100 ranked lists, with mAP averaged across 20 queries for each landmark category.
- The baselines include CNN-F variants, collaborative CNN variants, query-expansion methods, hypergraph modeling, and conventional feature-matching approaches.
- AQE averages retrieved-region Bag-of-Words vectors with the query, whereas DQE additionally uses lower-scoring photos as negative training data.
- PQE combines top-K retrieved photos with a query to identify patterns, while PAMQE uses LDA over an expanded multi-query set and user-photo matrix.
- MMHG models associations between landmark photos through multiple hypergraphs representing high-order relationships from different aspects.
4.4 Tuning Parameters
The study tunes the multi-query count, latent-topic threshold, and latent-factor dimension, then fixes their values for fair comparison and compact retrieval.
- Three parameters are tuned: K for multi-query-set size, λ for latent-topic selection, and L for latent-factor dimension.
- K=40, λ=0.4, and L=64 are selected following prior work for a fair comparison.
- The method sets s=20 to retain the 20 photos most similar to the query landmark in C-CNN feature space.
4.5 Comparison with Baselines
The baseline comparison evaluates deep feature representations on Flickr and Picasa, with C-CNN outperforming CNN-F variants and emphasizing informative landmark regions.
- The experiment uses ImageNet-pretrained CNNs and Caffe, while C-CNN training images are grouped into pseudo labels.
- Oxford Building images are used to construct quantization because their scenes resemble, but are not identical to, the test landmarks.
- C-CNN outperforms CNN-F, CNN-F+Average, and CNN-F+FV on Flickr and Picasa classification accuracy.
- User-informed pseudo-label fine-tuning helps construct the multi-query set and generate more discriminative features.
- For London clock towers, the strongest C-CNN responses concentrate on the tower pin and circular clock plate.
4.6 Cross Dataset Generalization
Cross-dataset generalization is evaluated across ImageNet, Places, Flickr, and Picasa, where dataset bias affects feature performance. ImageNet-pretrained CNN-F features fine-tuned on the target dataset perform best when training and testing use the same dataset.
- Cross-dataset generalization is affected by differences in data distribution, density, and diversity across ImageNet, Places, Flickr, and Picasa.
- ImageNet-pretrained CNN-F features fine-tuned on the target dataset achieve the best performance when training and testing use the same dataset.Places uses the learned parameters directly because it is scene-centric.
4.7 CNN Feature Training
CNN-F training uses smaller learning rates for fine-tuned hidden layers to control overfitting on the smaller training dataset, while retaining a 4096-dimensional FC2 representation.
- Lower initial learning rates for the fine-tuned FC1 and FC2 layers control overfitting on the smaller training dataset.The schedule changes from (10^-1, 10^-4) to (10^-5, 10^-5).
- The network's last hidden layer, FC2, has dimensionality 4096.
4.8 Comparison with State-of-the-art Approaches
The method is evaluated with precision-recall curves and mAP on Flickr and Picasa using sampled landmark queries. It outperforms competing approaches, with multiple-query methods benefiting from complementary information and high-level features.
- The evaluation samples 20 query photos per landmark in Flickr and Picasa and measures precision-recall and mAP.Duplicate photos are treated as junk in the mAP protocol.
- The proposed method outperforms competing approaches in precision-recall results, especially PAMQE.The authors attribute this to high-level deep features learned over the user-landmark latent subspace.
- Our method and PAMQE outperform all competitors because they exploit complementary information from multiple queries.The proposed method exceeds PAMQE through high-level features rather than mid-level pattern representations.
- DRLR degrades when a query photo is shot from a bad viewpoint, while MMHG underperforms while using low-level features.
4.9 Comparison with Query Expansion
The proposed query expansion exploits latent topics and user information instead of relying only on low-level matching or manual selection. It outperforms established query expansion methods in mAP on two databases.
- The proposed multi-query selection complements a query user by exploiting latent topics and user information.This differs from AQE and DQE's low-level feature matching and PQE's manual selection.
- PAMQE outperforms AQE, DQE, and PQE by a large margin in mAP on two databases.
- The proposed C-CNN with Fisher vector encoding outperforms PAMQE and produces highly discriminative features for landmark retrieval.
5 Conclusions
The paper proposes collaborative deep networks over landmark latent factors for robust landmark retrieval. Experiments on real-world social landmark photo datasets with user information report state-of-the-art performance against low-level and mid-level representation methods.
- The method achieved state-of-the-art performance against low-level feature and mid-level pattern representation methods on real-world social landmark photo datasets with user information.