Source-linked AI summary
Measuring Personalization of Web Search
Anikó Hannák, Piotr Sapieżyński, Arash Molavi Khaki, David Lazer, Alan Mislove, Christo Wilson
TL;DR
Web-search personalization can produce different results for the same query, raising concerns about Filter Bubble effects while remaining difficult to quantify. The paper develops a noise-controlled methodology and applies it to Google, Bing, and controlled account experiments. It reports measurable personalization associated with login status and geographic area, alongside 11.7% of Google results and 15.8% of Bing results differing due to personalization.
Problem
Negative effects and the extent and user attributes of deployed Web-search personalization have been insufficiently quantified.
Method
The paper compares repeated, controlled searches across real users and accounts with systematically varied features while controlling for temporal, geographic, infrastructure, and baseline noise.
Results
11.7% of Google Web Search results and 15.8% of Bing Search results showed differences due to personalization, with measurable effects triggered by login status and geographic area.
Takeaways & Limitations
The methodology provides a first step for measuring changing personalization practices, while the study finds few user properties associated with measurable personalization beyond login and location.
Takeaways & Limitations
Google comparisons used HTTP for AMT users and HTTPS for controls, and HTTP results disable some types of personalization.
Abstract
from arXiv · showhide
Web search is an integral part of our daily lives. Recently, there has been a trend of personalization in Web search, where different users receive different results for the same search query. The increasing level of personalization is leading to concerns about Filter Bubble effects, where certain users are simply unable to access information that the search engines' algorithm decides is irrelevant. Despite these concerns, there has been little quantification of the extent of personalization in Web search today, or the user attributes that cause it. In light of this situation, we make three contributions. First, we develop a methodology for measuring personalization in Web search results. While conceptually simple, there are numerous details that our methodology must handle in order to accurately attribute differences in search results to personalization. Second, we apply our methodology to 200 users on Google Web Search and 100 users on Bing. We find that, on average, 11.7% of results show differences due to personalization on Google, while 15.8% of results are personalized on Bing, but that this varies widely by search query and by result ranking. Third, we investigate the user features used to personalize on Google Web Search and Bing. Surprisingly, we only find measurable personalization as a result of searching with a logged in account and the IP address of the searching user. Our results are a first step towards understanding the extent and effects of personalization on Web search engines today.
1 INTRODUCTION
The paper addresses limited empirical knowledge about Web-search personalization and its possible Filter Bubble effects. It develops a controlled measurement methodology and applies it to Google, Bing, and user-feature experiments.
- Motivation: Personalized rankings may hide potentially important results, while the negative effects of personalization remain less understood than its benefits.Search engines generally do not identify which results were personalized or explain why they were selected.
- Contribution: The methodology controls for temporal index changes, distributed-index inconsistencies, and provider A/B tests when attributing result differences to personalization.The authors also provide a command-line implementation for the research community.
- Evaluation: 300 users with active Google and Bing accounts ran standardized searches while the study controlled for time, location, distributed infrastructure, and noise.The measurements covered Google Web Search, Bing Search, and DuckDuckGo.
- Findings: 11.7% of Google Web Search results and 15.8% of Bing Search results showed differences attributed to personalization.Personalization probabilities varied by query and tended to be higher for results toward the bottom of the page.
- Feature analysis: The study investigates profile information, browser and operating-system choice, search history, click history, and browsing history as possible personalization features.The authors create accounts with systematically varied behaviors to test these features.
120 search queries
The study uses search queries selected to provide broad topical coverage and to represent popular searches. Its broader conclusion is that measurable personalization is concentrated in login status and machine location.
- Observed features: Measurable personalization occurred for users who were logged in and for different machine IP addresses.The finding is reported for both Google and Bing.
- Observed features: All other tested user attributes produced differences no greater than baseline noise.This contrasts with the study’s examination of profiles, browser and operating-system choices, histories, and related behaviors.
- Scope: The methodology is presented as a first step that can be reapplied as search engines change their personalization techniques.The authors also state that it can generalize to other search services.
2 BACKGROUND AND RELATED WORK
The background describes search personalization as a long-studied technique with user benefits but limited transparency about how major deployed engines personalize results. Related work spans personalization methods, geographic and demographic signals, and measurement of deployed services.
- Search personalization: Google and Bing personalize search using mechanisms including location, account status, history, and social connections, while DuckDuckGo is designed not to personalize results.DuckDuckGo is included as a baseline in the study.
- Related work: Prior research reports strong performance from mining click histories, lower utility for user profiles, and query-dependent benefits of personalization.Other studies examine geographic location, inferred demographics, privacy-preserving search, targeted advertising, and discriminatory pricing.
- Measurement gap: Google’s public description identifies language, geolocation, query history, and Google+ connections as personalization signals, but specific uses of search history remain unclear.The cited description leaves open how temporal ordering and clicks affect personalization.
- Measurement gap: Existing work on comparing search engines identifies differences between engines but does not examine the impact of personalization.Other emerging work seeks to infer the features providers use to personalize deployed services.
3 METHODS
The methods compare controlled, repeated searches across accounts and engines while separating personalization from ordinary result variability. The design also measures result formats, query categories, and Google’s carry-over behavior.
- Result representation: Search-result pages are represented through their primary links, with static descriptive strings used when a result has no primary link.Google and Bing pages generally contain about 10 results, while DuckDuckGo analysis uses its top 10 organic results and ignores the contextual box.
- Experiment design: Each experiment varies one account feature, executes identical queries repeatedly, saves the results, and compares result identity and order.The design uses x accounts, q queries, and d days as its high-level parameters.
- Noise control: The study controls noise from search-index updates, geographically distributed infrastructure, and other sources of result variation.Queries are sent through standard search webpages and executed in lock-step across machines.
- Carry-over control: The carry-over effect was observed on Google but not on Bing or DuckDuckGo.Google-focused experiments therefore waited at least 11 minutes between subsequent searches.
4 REAL-WORLD PERSONALIZATION
A controlled AMT study compares real users’ search results with fresh-account controls to isolate personalization from search-result noise. It finds substantial, query- and rank-dependent personalization on Google and Bing, with location-specific variation especially common on Google.
- Collecting Real-World Data: The proxy recorded users’ HTML results, generated fresh-account controls, ran queries in parallel, and used fixed search-engine IP addresses to reduce noise.Control accounts had minimal search history, enabling comparison with real users whose providers had collected more data.
- Collecting Real-World Data: 300 AMT workers ran standardized searches through a controlled HTTP-proxy study targeting Google and Bing accounts.The sample included 200 Google participants and 100 Bing participants; users had to be U.S.-based and logged in during the study.
- Results: Top-ranked results were less personalized than lower-ranked results on both search engines.The analysis compares changes between AMT results and controls against the control-to-control noise floor.
- Results: Personalization varied by query: Bing’s most personalized queries concerned major political issues, while Google’s involved companies and politics.Factual “what is” and health-related queries were often among the least personalized on both engines.
- Results: Google personalization was often location-based, producing results for different store locations despite routing AMT requests through the same proxy IP address.This pattern was especially common for company-name searches.
5 PERSONALIZATION FEATURES
The experiments test whether login state, cookies, location, user profiles, and behavioral histories personalize search results on Google and Bing. Measurable differences appear mainly for login/cookies and geolocation, while profile and history treatments generally show little or no personalization.
- Experimental design: Fresh test and control accounts isolate personalization by comparing accounts that differ in one user feature.The methodology creates x + 1 accounts for each feature, including one control account, and compares their result lists with Jaccard Index.
- Login and cookies: Google’s login state and cookies change result ordering, with logged-in users receiving about two reordered results on average relative to users with no cookies.Google’s result sets are less affected than their ordering, while Bing cookies affect both returned results and result order.
- Other features: User-agent strings do not personalize results, while DuckDuckGo results remain essentially identical across IP addresses.The browser experiments show no clear differentiation from controls, and the IP-address result matches DuckDuckGo’s stated non-personalization policy.
- Geolocation: Google and Bing personalize results by geolocation: location tests have average Jaccard Index values of 0.91 on Google and 0.87 on Bing.Different locations generally differ by one result, with location effects even more pronounced when result order is considered.
- Geolocation caveat: Anomalous EC2 results demonstrate that incorrect IP geolocation can produce country-specific results independently of the Google account.Because the cause could not be determined and the effect affected only some machines, EC2 machines were excluded as tunnel endpoints.
- Profile and history features: The experiments find no measurable gender, age, zip-code, click-history, or browsing-history personalization on Google or Bing.Search-history accounts remain nearly identical to controls over 30 days, although longer experiments could potentially reveal larger differences.
6 QUANTIFYING PERSONALIZATION
Search-result personalization varies by query category and rank, with personalized pages changing more than non-personalized pages. The engines are conservative at rank 1, but differ in how they replace top results and embed their own services.
- Personalization of Query Categories: 88% of “what is” results are identical on Google, compared with 66% for “gadgets”; “politics” is the most personalized category on both engines.“Places” and “gadgets” follow “politics” as the most personalized categories.
- Personalization and Result Ranking: Personalized pages show more than twice as many result changes as non-personalized pages, with volatility peaking at rank 7 on Google and rank 8 on Bing.Changes peak at 33% on Google and 26% on Bing; rank 1 is least volatile.
- Personalization and Result Ranking: Only 5% of personalized rank 1 results change on Google and 3% on Bing, and most changes are swaps between ranks 1 and 2.The rank-1 analysis examines where the promoted result originated and where the former rank-1 result moved.
- Personalization and Result Ranking: Google and Bing differ when rank 1 changes: Google usually promotes a first-page result, while Bing more often inserts a completely new link.On Google, 93% of new rank-1 results come from the first page; Bing frequently inserts links to its own services.
- Personalization and Aggregated Search: Bing embeds its own services on 19% of result pages versus 9% for Google, while Google’s personalization raises service links from 8% to 12%.Google embeds services at rank 1 on approximately 15% of pages; Bing never does so at rank 1.
- Personalization and Aggregated Search: News links are not personalized in Google or Bing search results, even when other results on the page are personalized.News-link Jaccard Index and Kendall Tau coefficients remain approximately 1 across account characteristics.
7 CONCLUDING DISCUSSION
The paper quantifies personalization in Web search, identifies measurable triggers, and frames the findings as an initial step toward greater transparency. It also defines important boundaries: the study covers selected U.S. English queries, detects only positive instances, and measures literal rather than semantic differences.
- Little prior work quantified the basis and extent of personalization in Web search, despite concerns that personalization may hide information from users.
- The methodology measures personalization while controlling for numerous noise sources, and it was applied to real Google and Bing accounts.
- Measurable personalization on Google Search and Bing was triggered by logged-in accounts and requests from different geographic areas.
- The results are presented as a first step toward transparency, with future work proposed across additional search categories, devices, behaviors, and services.
- The study covers U.S. versions, English queries, and topics primarily relevant to U.S. residents, while identifying only positive personalization instances from finite tested terms.
- The analysis measures literal result differences, not whether differing pages contain different information or improve users’ search experiences.