Source-linked AI summary
Comparative Deep Learning of Hybrid Representations for Image Recommendations
Chenyi Lei, Dong Liu, Weiping Li, Zheng-Jun Zha, Houqiang Li
TL;DR
Image recommendations must bridge image semantics and user preferences, which are difficult to capture in separate representations. The paper learns shared image-user representations with a dual-net model and trains them comparatively using positive and negative images. Its experiments report better performance than naive learning and state-of-the-art image-recommendation methods.
Problem
Method
Results
Takeaways & Limitations
The experiments support using shared hybrid representations and comparative learning for personalized image recommendation.
Source: §7 · p. 8Some recommendations cannot be confidently judged against ground truth because users may not have seen the images.
Source: §6.3 · p. 8
Abstract
from arXiv · showhide
In many image-related tasks, learning expressive and discriminative representations of images is essential, and deep learning has been studied for automating the learning of such representations. Some user-centric tasks, such as image recommendations, call for effective representations of not only images but also preferences and intents of users over images. Such representations are termed \emph{hybrid} and addressed via a deep learning approach in this paper. We design a dual-net deep network, in which the two sub-networks map input images and preferences of users into a same latent semantic space, and then the distances between images and users in the latent space are calculated to make decisions. We further propose a comparative deep learning (CDL) method to train the deep network, using a pair of images compared against one user to learn the pattern of their relative distances. The CDL embraces much more training data than naive deep learning, and thus achieves superior performance than the latter, with no cost of increasing network complexity. Experimental results with real-world data sets for image recommendations have shown the proposed dual-net network and CDL greatly outperform other state-of-the-art image recommendation solutions.
1. Introduction
Image recommendations require representations that capture both image semantics and user preferences, which are difficult to express together using visual features alone. The paper proposes a dual-net model for shared image-user representations and comparative training, reporting gains over naive learning and state-of-the-art methods.
- Image recommendations are challenging because images lack an immediately effective representation and user intents are difficult to express through low-level visual features.Source: §1 · p. 1
- Deep networks offer automated image-representation learning, and the paper applies this approach to jointly represent images and user preferences.Source: §1 · pp. 1–2
- The dual-net maps images and user preferences into one latent semantic space, making their hybrid representations directly comparable for recommendation decisions.Source: §1 · pp. 1–2
- The comparative deep learning method trains on a user and positive/negative image pairs, learning their relative distances rather than a single user-image distance.After training, only the user and image sub-networks are needed.Source: §1 · p. 2
- The proposed solution outperforms naive learning and other state-of-the-art image recommendation methods in experiments on real-world datasets.Source: §1 · p. 2
2. Related Work
Prior work develops image representations and personalized recommendation methods, but the related-work review identifies joint modeling of image content and user preferences as an unresolved challenge.
2. Related Work
- The review frames the work around learning image representations and personalized image recommendations.Source: §2 · p. 2
2.1. Learning of Image Representations
- Deep learning automates image-feature extraction, while prior work also uses intermediate network outputs as image representations.Examples include supervised hashing and classification networks.Source: §2.1 · p. 2
- Earlier image-representation work focuses mainly on visual features, while jointly capturing image content and user preferences remains a challenging open problem.Source: §2.1 · pp. 2–3
2.2. Personalized Image Recommendations
- Personalized image-recommendation methods using social data often model users without image visual information, leaving user and image representations isolated.Source: §2.2 · p. 3
- Existing joint user-image approaches often process user and image information separately before combining them, motivating investigation of a fully integrated solution.Source: §2.2 · p. 3
3. Problem Formulation of Comparative Learning
Comparative learning addresses sparse and unbalanced implicit feedback by training mappings that place a user's preferred image closer than a non-preferred image in a shared latent space.
- The problem arises because users rarely rate abundant online images, and implicit feedback is sparse and severely unbalanced, with almost no negative feedback.Source: §3 · p. 3
- Comparative learning uses user-image-image triplets to learn relative user-image distances, addressing imperfect training data where naive single-distance learning may fail.Source: §3 · p. 3
- The objective treats the comparison as binary classification and uses cross-entropy to encourage a more discriminative distance difference.Source: §3 · p. 3
- The formulation assumes a simple distance function, such as Euclidean distance, because the focus is learning user and image representations.Source: §3 · p. 3
- The network represents users and images with three sub-networks and compares the resulting distances before applying its final cross-entropy loss.The two image-processing sub-networks are identical.Source: Figure 2 · p. 4
4. Comparative Deep Learning (CDL)
The CDL network compares user and image representations in a shared latent space. Its triplet-based architecture learns relative user-to-image distances, using image and user sub-networks and a weighted distance calculation.
4. Comparative Deep Learning (CDL)
- Triplet inputs compare a query user’s distances to a positive and a negative image; shared-parameter CNNs process images while a fully connected network processes user information.The image CNNs have identical configurations and shared parameters.Source: §4 · p. 4
- Image and user mapping functions produce 1024-dimensional representations in the same target latent space, making the two modalities comparable.The image network takes RGB pixel data as input; the user network takes a numerical user vector.Source: §4 · pp. 4–5
- A weighted l2-norm distance sub-network compares representations by weighting squared element-wise differences, with dropout at rate 0.5.The difference between the two distances is fed to a final cross-entropy loss and verified against the label.Source: §4 · pp. 4–5
5. CDL for Image Recommendations
For image recommendations, the paper converts user interests into cluster-based vectors and constructs training triplets using favorites and social data. Recommendations are selected from tagged candidates by user-image distance.
5. CDL for Image Recommendations
- The recommendation application focuses on suggesting new images based on a user’s browsing history and identifies user-vector generation and triplet preparation as prerequisites.The paper restricts its discussion to this setting rather than practical implementation details.Source: §5 · p. 5
- User tags are embedded with word2vector and clustered by k-means into 1,024 semantic clusters, which serve as bag-of-words dimensions for user vectors.This approach addresses sparse tag vectors and synonym handling; the clusters have varying frequencies and some topical polysemy.Source: §5 · p. 5
- Figure 3 plots the number of users interested in each of 1,024 clusters; users can be represented as bags of cluster words.The figure’s x-axis is clusters and its y-axis is user counts.Source: Figure 3 · p. 5
- Negative images are chosen from images without the user’s interest tags and outside friends’ favorites or images liked by users in the user’s groups.Favorites and likes provide positive examples; random sampling selects training triplets from the abundant negatives.Source: §5 · p. 5
- Recommendations are the K nearest candidate images with minimum user-image distances, after candidates are restricted to images sharing at least one user-interest tag.Representations may be calculated and stored in advance or calculated in parallel.Source: §5 · p. 5
6. Experiments
The experiments evaluate CDL hybrid representations on personalized image recommendation, compare them with alternative methods, and examine a case study for qualitative insights.
6.1.1 Datasets
- The Flickr dataset contains 101,496 images and 54,173 users; filtering yields 8,616 training users and 15,023 test users, with 20 concealed favorites per user.Training also uses 20 sampled triplets per user; testing mixes concealed favorites with 80 other images.Source: §6.1.1 · p. 6
6.1.2 Compared Approaches
- Comparisons include Borda Count with SIDL and visual-feature or metric-learning variants, plus TwoNets as the naive deep-learning baseline; CDL tests cross-entropy against hinge loss.TwoNets trains on user-image doublets with binary positive/negative labels, whereas CDL compares two images against a user.Source: §6.1.2 · p. 6
6.1.3 Implementation
- Images were resized to 256 × 256, and training took about four days on one GeForce Tesla K20c GPU with 5GB memory.The implementation used Caffe, dropout probability 0.5, learning rate 0.001, momentum 0.9, batch size 128, and weight decay 0.0005.Source: §6.1.3 · p. 6
6.2. Overall Performance
- CDL gains 42.58% in average precision and 46.50% in average recall over SIDL, leads for every K, and significantly and consistently outperforms TwoNets.Cross-entropy also has an obvious advantage over hinge loss in this task.Source: Figure 5 · p. 7, Figure 6 · p. 7, §6.2 · p. 7
6.3. Case Study and Insights
- Some recommendations marked incorrect may be unseen rather than disliked, and finer discrimination of similar objects may improve accuracy.The case study identifies a skeleton-like object as a likely mistaken recommendation despite its darker background.Source: §6.3 · p. 8
- Positive-image representations resemble one another and differ from negative-image representations, while sparse user inputs become dense learned representations through weighted L2 distance.The user input is a bag-of-words vector over clusters of word2vector results.Source: §6.3 · p. 8, Figure 9 · p. 8
7. Conclusions
The paper learns hybrid representations of images and user preferences in a shared latent semantic space for personalized recommendations. Its comparative learning approach performs better than naive deep learning and several state-of-the-art solutions in the reported evaluations.
- The approach learns hybrid representations that capture image information and users’ intents or preferences for personalized image recommendations.Source: §7 · p. 8
- A dual-net deep network maps users and images into a shared latent semantic space, where triplets of users and positive/negative images train relative-distance comparisons.Source: §7 · p. 8
- The empirical evaluations report better performance than naive deep learning and several state-of-the-art image recommendation solutions.Source: §7 · p. 8
- The authors propose applying comparative deep learning to other user-centric applications, including image search and image editing, as future directions.Source: §7 · p. 8