Source-linked AI summary
Visual Search at Pinterest
Yushi Jing, David Liu, Dmitry Kislyuk, Andrew Zhai, Jiajing Xu, Jeff Donahue, Sarah Tavel
TL;DR
The paper addresses the challenge of deploying commercially useful visual search with limited engineering and computational resources, amid few published end-to-end commercial architectures. It presents a scalable pipeline using widely available tools and evaluates it in Pinterest applications, reporting improved visual-search accuracy and significant user-engagement gains.
Problem
Commercial visual search requires scalable, cost-effective deployment and live validation, while few publications describe end-to-end architectures for real-world applications.
Method
The paper builds an end-to-end Pinterest pipeline with distributed incremental feature extraction, real-time search, metadata-assisted two-step object detection, and application-specific evaluations.
Results
Visual search significantly improved search accuracy and user engagement across Pinterest applications, with less than 1% false positive rate and VGG features improving benchmark accuracy.
Takeaways & Limitations
Widely available distributed platforms and open-source tools can support a large-scale visual search system built and operated by a small engineering team.
Takeaways & Limitations
The authors identify CNN-based object detection performance and efficiency, use of the curation graph, and alternative visual-search interfaces as areas for further work.
Abstract
from arXiv · showhide
We demonstrate that, with the availability of distributed computation platforms such as Amazon Web Services and open-source tools, it is possible for a small engineering team to build, launch and maintain a cost-effective, large-scale visual search system with widely available tools. We also demonstrate, through a comprehensive set of live experiments at Pinterest, that content recommendation powered by visual search improve user engagement. By sharing our implementation details and the experiences learned from launching a commercial visual search engines from scratch, we hope visual search are more widely incorporated into today's commercial applications.
1. INTRODUCTION
Pinterest presents a scalable, cost-effective visual search system designed for a small engineering team and evaluates it in two commercial applications. Live experiments show that visual search supports object-based discovery, expands recommendations, and improves user engagement.
- Motivation: Visual search addresses a gap in published end-to-end architectures for commercially deployed systems, whose complexity and proprietary nature limit disclosure.The paper frames this gap against the growth of online photos and search engines.
- Approach: The system targets deployment cost and scalability through widely available tools, distributed processing, incremental feature updates, and real-time search infrastructure.Its object-detection approach also uses category classification before per-category detection to reduce expensive computation.
- Applications: Related Pins uses visual search to recommend Pins for the long tail of content lacking recommendations from the existing curation graph.The application supplements graph-based recommendations with visual signals.
- Applications: Similar Looks localizes objects such as bags and shoes so users can select an item and explore visually similar products rather than searching the whole image.The system computes similarity between localized query objects and database images.
- Results: Visual search experiments achieved less than 1% false positive rate with good detection, improved benchmark accuracy using VGG representations, and produced significant user-engagement gains.The paper evaluates object detection, feature representations, and end-to-end applications through component tests and live A/B experiments.
2. VISUAL SEARCH ARCHITECTURE AT PINTEREST
Pinterest’s visual search architecture combines metadata-guided detection, deep and color features, distributed incremental computation, and real-time search services to index very large image collections cost-effectively. Experiments report low false-positive detection, improved click prediction from CNN representations, and infrastructure that avoids repeated feature computation.
- Pinterest propagates annotations through its user-board-image graph, providing semantic information that supports image representation and visual search.
- The architecture extracts local, deep, and salient-color features, with CNN representations binarized for efficient Hamming-distance comparison.Deep features use AlexNet- and VGG-based CNNs, including fc6 and fc8 representations.
- Image Representation and Features: Textual metadata first predicts likely image categories, after which only category-specific object detectors run, reducing computation and false positives.The two-step pipeline uses aggregated pin descriptions and board titles to select relevant detectors.
- Two-step Object Detection and Localization: Less than 1% false positive rate was achieved with good detection rate by combining object detection and localization with metadata.This performance was described as vital for the application.
- Click Prediction: CNN-based click prediction outperforms the PHOW-SVM baseline for both close-up rate and click-through rate, with end-to-end fine-tuning providing an additional boost.The models predict user interaction probabilities from visual features for ranking, recommendations, and ad targeting.
- Incremental Fingerprinting Service: The incremental fingerprinting service updates features for new images or feature changes without recomputing unchanged images.An initial full computation takes a little over a day on several hundred 32-core machines and produces roughly 5 TB, while steady-state updates require about 5 machines.
3. APPLICATION 1: RELATED PINS
Related Pins traditionally relied on curated image-to-board relationships and content signals, leaving some less popular or newly created Pins without recommendations. Visual Related Pins uses visual signals to provide recommendations for those images, including near-duplicate retrieval through a local token index.
- 6% of Pinterest images had very few or no recommendations because traditional recommendations required existing curation.Visual Related Pins was introduced for these images using the visual search pipeline and visual signals.
- Visual Related Pins recommends images related to the Pin a user is currently viewing, extending Related Pins beyond curated image-to-board relationships.
- The product first uses a local token index to identify near duplicates and transformed versions of the query image.Supported transformations include resizing, cropping, rotation, translation, and minor content changes.
Static Evaluation of Search Relevance
The evaluation compares CNN feature representations for visual-search relevance using Pinterest image annotations and precision@k. VGG 16-layer FC6 features produced a substantial precision gain with acceptable service latency.
- Evaluation setup: Pinterest annotations served as proxy relevance labels for an offline visual-search dataset of about 1.6 million unique images.The dataset retrieved 3,000 images for each of 1,000 text queries.
- Feature comparison: The comparison evaluated original and fine-tuned AlexNet, GoogLeNet loss3, and VGG 16-layer FC6 feature sets using precision@k.The study also examined combining low-level feature scores with semantic classifier outputs.
- Results: VGG 16-layer FC6 features delivered a substantial precision gain against the evaluation dataset with acceptable visual-search latency.Table 1 reports p@5, p@10, and average service latency for the compared models.
Live Experiments
The live Related Pins experiment replaced insufficient recommendations with visual-search results for eligible Pin close-ups and measured engagement through repins. After three months, the treatment increased total Related Pins repins by 2%.
- Experiment setup: The system detected new Pins with few recommendations, queried visual search, and stored results in HBase for Pin close-up serving.This prepared recommendations before they were served in the live product.
- Experiment setup: The experiment traded recall for relevancy by applying a metadata conformity threshold to reduce poor recommendations.The treatment replaced Related Pins with visual-search results for eligible users, while control users saw the unchanged experience.
- Results: 2%: Visual Related Pins increased total repins in the Related Pins product after three months.The experiment initially ran on 10% of eligible live traffic, and repins were selected as the engagement metric.
4. APPLICATION 2: SIMILAR LOOKS
Similar Looks addresses fashion Pins that often lack actionable shopping links by detecting individual products and recommending visually and semantically similar items. Users access these recommendations through object-level interactions in the Pin.
- Motivation: Women’s fashion Pins often contain editorial imagery and multiple objects, while many do not direct users to actionable shopping experiences.User research identified this as a common frustration, and users clicked through less often than for other categories.
- Method: Similar Looks localizes and classifies fashion objects such as bags, shoes, pants, and watches in Pin images.Object recognition supplies the product-level regions used for recommendation generation.
- Method: The system extracts visual and semantic features from detected objects to generate product recommendations displayed through red-dot interactions.Clicking a red dot loads a feed of Pins featuring similar products.
Related Work
Visual search for fashion recommendations has been pursued by both academic and industry systems. The paper situates its object-based approach among commercial applications and growing vision-based fashion-recommendation research.
- Industry systems: Commercial examples of computer vision for fashion recommendations include Like.com, Google Shopping, and Zappos.Baidu and Alibaba are also described as having launched related visual-search systems.
- Academic research: Vision-based fashion recommendations represent a growing research area.The passage cites multiple prior research efforts in this area.
- Positioning: The paper’s approach is presented as an object-based visual-search system for fashion recommendations.It is discussed alongside related academic and industrial applications.
Static Evaluation of Object Localization
The evaluation combines text metadata with image-based object localization to detect fashion objects more accurately and with fewer false positives. Performance varies by object class, making the combined approach especially useful for difficult categories.
- Dataset: The evaluation dataset contains 2,399 manually labeled fashion objects across 9 categories sampled from Pinterest women’s fashion images.Labels were rectangular object crops used to evaluate text filtering, image-based detection, and their combination.
- Detection performance: Combining text filters with object localization reduced the false positive rate to less than 1%, versus 6.7% for text filters and 2.5% for localization alone.The combined pipeline applies text filters before object detection.
- Class-specific behavior: Text annotations were insufficient for glasses, where image-based classification performed better because the category has a distinctive visual shape.The results show that the relative value of metadata and visual evidence depends on the object class.
- Class-specific behavior: Dress detection had a 12.3% false positive rate, but adding a text filter dramatically improved results under occlusion and high style variation.Text prefiltering was also crucial for achieving a false positive rate of 1% or less for shoes, bags, and pants.
Live Experiments
Pinterest evaluated Similar Looks through live traffic by measuring interactions with detected-object dots and visually similar results. Initial dot engagement was positive, but Similar Looks reduced total pin-close-up engagement when compared with existing related content.
- Live deployment: The system identified over 80 million clickable objects and measured engagement after launching to a small percentage of Pinterest live traffic for one month.Metrics included clicks on object dots, clicks on visual search results, and comparison with Related Pins.
- Similar Looks engagement: 12% of users who viewed a pin with a dot clicked on a dot on an average day.Users subsequently clicked an average of 0.55 Similar Look results.
- Overall engagement: Similar Looks reduced overall pin-close-up engagement by 4% when its engagement was combined with Related Pins and compared with the control.The red-dot clickthrough rate gradually decreased after the novelty effect and stabilized around 10%.
- Experimental design: A blended experiment inserted Similar Looks results directly into Related Pins to measure recommendation relevance without bias from users learning the new object-dot behavior.This design compared visually similar recommendations with non-visual recommendations on pins containing detected objects.
5. CONCLUSION AND FUTURE WORK
The paper concludes that widely available distributed platforms and open-source tools can support a large-scale visual search system built by a small team. It also reports live product experiments and identifies further work in detection efficiency, relevance, and interface design.
- Conclusion: A handful of engineers or an academic lab can build a large-scale visual search system using distributed platforms and non-proprietary tools.The implementation includes incremental feature updating and a two-step object detection and localization method.
- Conclusion: Live product experiments demonstrate that visual search features can increase user engagement.The conclusion frames this as an outcome of deploying visual search in product experiments.
- Future work: Future work includes improving the performance and efficiency of CNN-based object detection methods for live visual search systems.The authors also plan to leverage Pinterest’s curation graph to improve relevance.
- Future work: The authors plan to experiment with alternative interactive interfaces for visual search.This is listed alongside planned improvements to detection and relevance.