Source-linked AI summary
Adscape: Harvesting and Analyzing Online Display Ads
Paul Barford, Igor Canadi, Darja Krushevskaja, Qiang Ma, S. Muthukrishnan
TL;DR
The paper studies the features, mechanisms, dynamics, and targeting of online display advertising at broad scale. The authors develop scalable profile-based crawling that gathers display ads, creatives, landing pages, and targeting data across websites and user profiles. Targeting was widespread, with most sites using it on over 80% of display ads, yet many ads reached all users; over 3.7K marketers were identified, and ads generally aligned with profiles.
Problem
The paper studies the features, mechanisms, dynamics, and targeting of online display advertising at broad scale.
Method
The authors develop scalable profile-based crawling that gathers display ads, creatives, landing pages, and targeting data across websites and user profiles.
Results
Targeting was widespread, with most sites using it on over 80% of display ads, yet many ads reached all users; over 3.7K marketers were identified, and ads generally aligned with profiles.
Takeaways & Limitations
The findings provide an empirical foundation for understanding the diversity of the Adscape and its targeting mechanisms.
Takeaways & Limitations
The study restricts data collection to English-language websites, fixes geolocation, and does not consider time-of-day effects.
Abstract
from arXiv · showhide
Over the past decade, advertising has emerged as the primary source of revenue for many web sites and apps. In this paper we report a first-of-its-kind study that seeks to broadly understand the features, mechanisms and dynamics of display advertising on the web - i.e., the Adscape. Our study takes the perspective of users who are the targets of display ads shown on web sites. We develop a scalable crawling capability that enables us to gather the details of display ads including creatives and landing pages. Our crawling strategy is focused on maximizing the number of unique ads harvested. Of critical importance to our study is the recognition that a user's profile (i.e. browser profile and cookies) can have a significant impact on which ads are shown. We deploy our crawler over a variety of websites and profiles and this yields over 175K distinct display ads. We find that while targeting is widely used, there remain many instances in which delivered ads do not depend on user profile; further, ads vary more over user profiles than over websites. We also assess the population of advertisers seen and identify over 3.7K distinct entities from a variety of business segments. Finally, we find that when targeting is used, the specific types of ads delivered generally correspond with the details of user profiles, and also on users' patterns of visit.
1. INTRODUCTION
This study characterizes the online display-ad landscape and its targeting mechanisms through profile-based crawling that gathers ads and associated data at scale.
- The study examines display-ad diversity, delivery dynamics, and targeting across websites and user profiles.
- Profile-based crawling treats each crawler instance as a distinct user, enabling collection of ads delivered to different browser profiles.
- Over 175K distinct ads and associated landing pages and creatives were gathered from 180 large English-language websites using 340 user profiles.
- Targeting was used on over 80% of sites’ ad inventory, yet many sites also delivered substantial numbers of profile-independent ads.
- The analysis identified over 3.7K marketers and found widespread interest-, age-, and gender-based targeting.
2. THE ANATOMY OF DISPLAY ADS
The Adscape consists of interacting advertisers, publishers, intermediaries, users, and allocation rules that determine which ads appear under particular conditions.
- Advertiser View: Advertisers combine creatives, placements, demographics, interests, timing, geography, and frequency caps to define campaigns.
- Advertiser View: Targeting strategies rely on players that track cookies and maintain user profiles.
- Publisher View: Publishers can source ads through premium campaigns, ad networks, and auction-based ad exchanges, often combining these methods on one page.
- The Adscape View: The Adscape models an allocation function mapping a user, website, and time to the ads shown.
- The Adscape View: This allocation may depend on user demographics and interests, site context, time and history, and constraints among available ads.
3. CHALLENGES IN OBSERVING THE ADSCAPE
Observing the Adscape is harder than ordinary web crawling because ads vary across user states, are dynamically delivered, can contaminate crawler profiles, and impose impression costs.
- Crawlers must identify ad elements that may be delivered through JavaScript and require multiple executions to retrieve.
- Ads can change across reloads and user conditions even when webpage content remains unchanged, making ordinary recrawl strategies insufficient.
- Because ads are critically targeted to user profiles, understanding the Adscape requires mimicking multiple personas rather than a single viewer.
- Ad visits can contaminate crawler profiles, while impression-based charging means limiting crawler activity also limits the observed sample.
- An ad crawler must explore pages across user profiles, geolocations, day parts, and other states, producing a state space at least 10^7 times larger under the paper’s assumptions.
4. OUR APPROACH
The approach restricts analysis to interest-based personalization and uses budgeted, profile-aware crawling to maximize the number of distinct ads collected from website–persona pairs.
- The study fixes geolocation and ignores time-of-day effects, focusing on how user profiles map website–persona pairs to ads.
- Short and long crawling strategies vary rapid sequential visits and repeated visits to expose different ad-delivery behavior.
- Website and persona pools are built from popular sites and diverse interests, respectively.
- The crawler selects a budget-limited focus set of website–persona pairs to maximize the union of distinct ads observed.
- The selection problem is NP-hard, so the method uses an approximate greedy maximum-cover algorithm.
5. PROFILE-BASED CRAWLING
The study builds profile-based crawling infrastructure to generate, preserve, harvest, classify, and parse ads for controlled user personas. It addresses profile contamination and identifies ads through browser-observed visual elements and multiple classification tests.
- Profile Generation: Profile-based crawling generates personas from interest categories and uses isolated browser profiles to study ads shown to controlled user types.The approach avoids server-side profiling and Flash cookies, and cannot discover retargeted ads without first visiting an advertiser’s site.
- Ad Harvesting: Firefly loads pages in a browser, captures visual elements, iframe and landing-page URLs, dimensions, and local copies of ad creatives.Browser execution is required because ads are often initialized by JavaScript during page load.
- Identifying Ads: An element is classified as an ad only if it matches an AdBlock rule, a standard dimension, and an external-link requirement.The system maintains 25 standard dimensions and excludes elements linking within the same domain.
- Uncontaminated Crawling: Profiles become contaminated as websites are visited, so the crawler limits visits to preserve the intended persona during experiments.After 50 visits, 60% of profiles gained more than 9 new interests; limiting visits to five or fewer mitigated the issue.
- Landing-Page Parsing: The parser resolves redirect chains to store each ad’s final landing-page destination for analysis.Intermediate redirects may support click accounting, but the study retains only the destination URL.
6. DATA COLLECTION
Data collection combines popular websites, diverse interest-based personas, controlled crawling strategies, and a budgeted focus set. The resulting corpus contains 875,209 impressions and 175,495 distinct ads, while automated collection introduces measurable missing-data and classification limits.
- Selecting W and P: The study selects 314 popular English-language ad-supported websites and 340 personas based on Google’s advertising-interest hierarchy.Personas were diversified using second-level interests and popular Google Display Network websites.
- Initial Profile Contamination: Initial profiles often contain irrelevant interests, and all profiles for third-level categories have fewer than 20% relevant interests.Health-related profiles frequently contain zero relevant interests, except for selected health categories.
- Crawling Strategies: New-ad arrival drops sharply around visit 5 and then decreases slowly, motivating short (10,5) and long (100,1) crawling strategies.The post-tenth-visit decline approximately follows y = −0.6x + 75; the two strategies trade shorter sessions against broader repeated exploration.
- Dataset: 875,209 impressions yielded 175,495 distinct ads from 3,700 advertisers served through 106 ad servers.The corpus was collected over two days from the final focus set.
- Sources of Missing Data: Automated collection timed out on 0.5% of page loads, while 1% of ads lacked usable landing-page connections and detection could produce false positives or negatives.The classifier favors reducing false positives, and ads without landing-page connections remain usable for most analyses but not landing-page or advertiser analysis.
7. EMPIRICAL RESULTS
The empirical study characterizes ad collection, targeting, advertiser diversity, and page-level ad allocation across websites and user profiles. Results show strong profile effects, concentrated website contributions, diverse advertisers, and mostly multi-advertiser pages.
- 175,495 distinct ads were harvested from over 875K impressions using 340 profiles, with distinct-ad arrival slowing to saturation approximately one day into crawling.The crawler collected ads over a relatively short period, and the number of new ads eventually grew more slowly than visits.
- Distinct-ad collection grew almost linearly with profiles, while 2% of websites produced 90% of observed distinct ads.This makes profiles broadly useful for expanding coverage, whereas many websites were redundant for distinct-ad harvesting.
- Multiple advertisers per page were the norm: 75% of two-placement pages used different advertisers, while three-placement pages used three advertisers 50% of the time.Exclusivity nevertheless occurred, including 628 six-placement pages containing ads from a single advertiser.
- Only 312 ads satisfied the empirical frequency-cap criterion, although most ads had small frequency caps; longer crawling strategies captured the largest number of distinct ads.Short strategies matched the distinct-ad yield of the first 50 visits of long strategies, but neither strategy could be excluded when maximizing collection.
- The corpus contained over 3.7K advertisers spanning categories including Financial Services, Shopping, Computers, Business, Arts & Entertainment, and Education.Advertiser impressions were highly uneven: more than 80% of advertisers had no more than 100 impressions, while the top advertiser accounted for about 10% of all impressions.
8. RELATED WORK
Prior work addresses web crawling, user-centric crawl scheduling, browser-history-informed URL organization, privacy, and ad filtering, but does not describe methods for crawling display ads.
- User-centric crawling has focused on scheduling pages and using browser-history hints to organize URLs, rather than harvesting display advertisements.
- The paper reports no prior work that builds and uses user profiles specifically for ad crawling, and no described methods for crawling display ads.
- Display-ad crawling differs from standard web crawling because ads can change on reload, depend on information beyond page context, and must be distinguished from other page visuals.
- Privacy studies show that targeted ads can reveal user-profile interests, while tracker taxonomies characterize the intermediaries involved in online advertising.
9. CONCLUSIONS
The paper introduces a profile-based infrastructure for collecting and analyzing online display ads at scale. Its corpus supports findings about targeting prevalence, advertiser diversity, and alignment between ads and user profiles, while future work expands coverage and mechanism analysis.
- Over 175K unique ads from more than 180 English-language websites and 340 user profiles form the basis for evaluating the Internet Adscape.
- The profile-based Firefox crawler distinguishes display ads from other images and collects creatives, landing pages, and related data at scale.
- Most visited sites use targeting mechanisms on over 80% of display ads, yet many also deliver ads to all users regardless of profile.The paper suggests that insufficient advertiser targeting specificity may explain profile-independent delivery.
- The study identifies over 3.7K unique marketers across diverse business segments and finds general alignment between delivered ads and user profiles.
- Future work will expand data collection, conduct longitudinal analyses, include video advertising, and model targeting mechanisms more deeply.