PaperlayerView exact revision on arXiv

Source-linked AI summary

What Users See - Structures in Search Engine Results Pages

Nadine Hoechstoetter, Dirk Lewandowski

arXiv:1511.05802v1cs.IR

TL;DR

Search-engine results pages combine organic results with advertisements and specialized shortcuts, but the complete composition visible to users has been insufficiently examined. The paper studies that composition by sending popular and rare queries to four major engines and counting their displayed elements. It finds substantial differences among engines and queries, while organic results remain dominant and shortcuts, Wikipedia, and engine-specific offerings also matter.

  • Problem

    Existing research does not examine search-engine results pages as complete presentations containing all available result elements.

  • Method

    The study empirically counts results-page elements across major search engines and models the elements visible to users.

  • Results

    Search engines use different results-page strategies, with sponsored listings, Wikipedia results, and engine-specific offerings distributed differently across engines.

  • Takeaways & Limitations

    Results-page composition shapes what users see and should be considered when evaluating search engines and retrieval effectiveness.

  • Takeaways & Limitations

    The study’s conclusions are constrained by its use of U.S. interfaces and may not apply to country-specific search interfaces.

Abstract

from arXiv · show

This paper investigates the composition of search engine results pages. We define what elements the most popular web search engines use on their results pages (e.g., organic results, advertisements, shortcuts) and to which degree they are used for popular vs. rare queries. Therefore, we send 500 queries of both types to the major search engines Google, Yahoo, Live.com and Ask. We count how often the different elements are used by the individual engines. In total, our study is based on 42,758 elements. Findings include that search engines use quite different approaches to results pages composition and therefore, the user gets to see quite different results sets depending on the search engine and search query used. Organic results still play the major role in the results pages, but different shortcuts are of some importance, too. Regarding the frequency of certain host within the results sets, we find that all search engines show Wikipedia results quite often, while other hosts shown depend on the search engine used. Both Google and Yahoo prefer results from their own offerings (such as YouTube or Yahoo Answers). Since we used the .com interfaces of the search engines, results may not be valid for other country-specific interfaces.

Abstract

Search engine results pages increasingly combine organic links with advertisements and results from specialized collections, making the visible result set a central object of evaluation. This study examines how major engines compose those pages and what users are presented.

  • User-visible results: Limited screen space and users’ reluctance to scroll make the first results screen especially important for what users see and select.The paper distinguishes the complete first results page from the immediately visible results screen.
  • Motivation: The paper frames SERP composition as relevant to search-engine quality and user satisfaction, alongside the visibility of paid listings.It treats advertisements and other non-organic elements as part of the results-page experience.
  • SERP composition: Search engines increasingly aggregate results from multiple collections, including news, images, video, and other specialized indexes.These additions occupy space alongside the core Web results list.
  • Aim: The paper investigates how search engines compose results pages from organic results, advertisements, and additional results from specialized collections.It focuses on the elements users encounter on the first page and visible screen.
  • Method: The study empirically counts different SERP elements to draw conclusions about what users encounter on major search engines.Its scope includes results presentation, user search behaviour, and the design of the investigation.

Search engine results pages

Search engine results pages combine organic listings with advertisements, shortcuts, snippets, prefetches, and other specialized results. Their layouts and available elements differ across engines, interfaces, screen areas, and countries.

  • Visible area: The visible area depends on screen and browser-window size, whereas the scrolling area contains material users must move down to see.The paper illustrates the distinction using a Google results screen.
  • SERP elements: Organic results form the central component of SERPs, but engines also integrate results from specialized collections and databases.These additions can include book, image, news, blog, patent, flight, and parcel-tracking results.
  • SERP elements: Shortcuts are specialized results shown above organic listings and triggered by the query or a special search term.They are also called one-box results or similar labels.
  • SERP elements: Snippets extend ordinary result descriptions with additional navigational links, while prefetches show a smaller set of links from the same site.Google and Yahoo use prefetches; snippets can include several site links or an additional search box.
  • Scope: The examples concern U.S. interfaces, and results, sponsored-link placement, and available snippets can differ across countries.The investigation is concentrated on the U.S. market.
  • Engine layouts: Google, MSN, and Ask.com place sponsored results and other elements in different positions and quantities on their first pages.The examples describe distinct arrangements of sponsored links, suggestions, organic results, and specialized results.

Literature review

Prior research examines user behaviour, result presentation, sponsored listings, and retrieval effectiveness, but largely focuses on selected SERP elements. The paper identifies a lack of studies treating the results page as a complete, integrated presentation.

  • User behaviour: Users commonly inspect only a few results and often remain on the first results page, making the visible presentation especially consequential.Studies report rapid evaluation, limited scrolling, and infrequent use of additional search collections.
  • Methodological limits: Logfile studies often report pages viewed but not the results selected or the order of clicks, requiring qualitative research to identify actual clicked positions.This limits what logfile data alone can establish about user selection.
  • Research coverage: Research on query refinement tools and other SERP elements is limited, and existing studies find only moderate use of refinement tools.The literature includes studies of AltaVista Prisma and Dogpile.com.
  • Sponsored results: Users generally prefer organic results over sponsored links, which they are more likely to ignore than click.In one study, participants went first to organic results on more than 80 percent of searches and to sponsored links on 6 percent.
  • Research gap: Earlier studies frequently concentrate on organic results rather than the full range of results-page presentation possibilities.The paper argues that understanding what users see requires the complete SERP picture.

Results presentation

As search engines add special-collection results and advertisements, organic results occupy a smaller share of the results-page presentation. The paper uses screen-space measures to compare how much room remains for organic results.

  • Changing presentation: Additional collection results and advertisements increasingly occupy space above organic results, reducing the prominence of organic listings.The paper describes this as a general change in results presentation.
  • Measurement: The editorial precision measure estimates the share of SERP screen space used by organic results.It divides total screen space by the space used for editorial results.
  • Comparison: The ratio of space devoted to organic results differs across search engines.This comparison treats page composition as an aspect of retrieval effectiveness evaluation.

Retrieval effectiveness studies

Earlier evaluations largely focused on organic results, sometimes sponsored results, and often assumed a dedicated searcher following links in order. This paper extends that perspective to all SERP elements and asks how their composition differs across search engines.

  • Prior studies mainly evaluated organic results, omitting sponsored results and additional collections.
  • Sponsored-versus-organic effectiveness research generally addressed queries with commercial purposes.
  • Traditional TREC-style evaluations model a dedicated searcher who follows links in presented order and clicks all results.
  • The paper continues work on how much space each result type occupies by accounting for all described SERP elements.
  • The study asks about sponsored-link counts, preferred hosts and content types, result-type differences, special displays, shortcuts, and cross-engine differences.

Data collection

The study sampled popular and rare queries, downloaded results pages, and classified their elements across selected search engines. Collection used automated HTML analysis, with country-specific interfaces and request blocking shaping the procedure.

  • The sample contained 500 popular and 500 rare queries drawn from the top and last 100k queries in an Ask.com query log.
  • Queries were sorted by frequency and alphabetically, then every second hundredth query was selected for representation across the list.
  • Scripts downloaded results pages, and engine-specific programs parsed HTML patterns to classify each result as organic, paid, snippet, or another type.
  • Yahoo blocking led the researchers to limit requests to 25 followed by a 2-hour timeout.
  • All engines used US web interfaces, while Google required a US proxy because sponsored links depended on the detected IP country.
  • Google sponsored-link URLs were unavailable because the proxy masked Adwords, leaving only their positions and counts.

Results

The results sets differed substantially across engines and query types, especially in sponsored-result prevalence and processing success. Popular queries generally produced more advertising, while rare-query processing was more vulnerable to failures or blocking.

  • 12,522 Google results came from 499 popular and 498 rare queries; only 16 rare queries produced no results.
  • 9,436 Yahoo results were processed, including 5,232 from popular and 4,204 from heavy-tail queries; 64 produced no results versus 2 popular queries.
  • 11,752 MSN results included all 500 popular-query results, but only 457 rare queries were valid because others were blocked or failed.
  • Ask.com produced 9,127 URLs: 5,224 from popular queries and 3,903 from rare queries, with five popular queries yielding empty sets.
  • Popular queries clearly received more sponsored links than rare queries.

Characterisation of the Organic Results Sets

Organic results remained the dominant SERP component, but engines differed in advertising placement, host preferences, file types, and Wikipedia prominence. Wikipedia appeared frequently across engines, while Google and Yahoo prominently featured their own services.

  • Wikipedia was the most popular host across all engines, and Google preferentially boosted it toward the first position.
  • More than half of hosts appeared only once: 54.7% for Google, 55.1% for Yahoo, 57.9% for MSN, and 58.1% for Ask.com.
  • HTML/HTM pages were the most common file types, while Ask.com showed fewer Microsoft Office documents and PDFs than competitors.
  • Organic results comprised 77% of Google URLs, 89.6% of Yahoo results, 78.4% of MSN/Live results, and 89.7% of Ask.com results.
  • Ads appeared above organic results for 38% of Yahoo and MSN queries versus approximately 27% for Google and Ask.com.
  • Google had at least one sponsored link for 297 popular-query SERPs and 230 rare-query SERPs.
  • Ask.com placed no advertisements to the right of organic results, instead showing them above and below the organic list.

Use of Shortcuts for Popular Queries

Search engines differ substantially in how they use shortcuts, with Google and Ask.com presenting broader or more frequent shortcut results than Yahoo and MSN/Live.

  • Google: Google produced 1.3 smart results per popular query and 0.9 per rare query when all queries were included.News results appeared in the third or tenth organic position, while snippets and prefetches were more common than shopping links.
  • Yahoo: Yahoo used few shortcut categories, including prefetches and snippets, and showed fewer results from other collections than the other engines.Its shortcut distribution was nearly the same for popular and rare queries.
  • MSN/Live: MSN/Live offered more shortcut categories than Yahoo, but they appeared infrequently, including for popular queries compared with Google or Ask.com.The passage reports a difference between popular and rare queries without quantifying it.
  • Ask.com: Ask.com had the largest shortcut portfolio, placed shortcut links at the top, and showed more shortcuts for popular queries.Ask 3D could automatically introduce results from other collections, creating possible redundancies such as several news results.
  • Google: Google used the most sophisticated results, flexibly positioning shortcuts and drawing on services such as Google Maps.The passages attribute this sophistication partly to Google’s knowledge of user preferences and its additional services.

Discussion

Search engines shape what users see on the first results screen by combining organic listings with shortcuts, multimedia, and promoted or affiliated sources. This composition affects evaluation, trust, and the information users encounter.

  • SERP composition: The first results screen is no longer composed solely of pure Web results but includes multimedia and other injected result types.Search engines also predict query needs and promote collections such as image and news searches.
  • Bias and neutrality: Google showed YouTube results disproportionately compared with other engines, illustrating how a search engine may favor its own offerings.The paper links this observation to broader concerns about the composition of visible results.
  • Bias and neutrality: Search engines are not neutral results lists because boosted results, shortcuts, optimized sites, and favored hosts shape the visible area.The discussion distinguishes legitimate high-quality hosts from problematic preference for a search engine’s subsidiaries.
  • Implications: Users should understand SERP composition before deciding how much to trust search results.The study frames this awareness as relevant to information and knowledge acquisition in a market dominated by a few engines.
  • Implications: Because prominent listings are often occupied by preselected hosts and Wikipedia, taking the first listings is especially difficult for popular terms.The paper identifies implications for search engine optimization.
  • Evaluation: Retrieval-effectiveness tests should count all visible SERP elements because additional collections or shortcuts may satisfy user needs.Restricting evaluation to organic results introduces bias.

Conclusion

Search engines differ substantially in how they compose results pages, including sponsored, special, and organic results. Query popularity also affects result composition, while Wikipedia and certain host preferences reveal engine-specific ranking and presentation patterns.

  • Sponsored results: Sponsored results are more prominent for popular queries, with Google showing the highest share of paid listings.Ask’s sponsored results are similar because it uses Google’s sponsored links.
  • Host preferences: Search engines show different preferences for hosts and top-level domains, although .com is preferred across engines.Popular hosts are reused across result pages, but the leading hosts differ by engine.
  • Host preferences: Wikipedia appears frequently across engines, but Yahoo and MSN show more Wikipedia results overall while Google commonly places Wikipedia first.Google therefore emphasizes prominent placement rather than the largest total number of Wikipedia links.
  • Indexed content: HTML/HTM is the most common file type across engines, indicating that their indexes remain primarily based on HTML content.Other file types are less prevalent because HTML provides more retrieval possibilities than images or videos.
  • Special results: Google leads in representing special results, with more varied result types than the other search engines.The frequency of these special-result types differs between rare and popular queries.

Further Research

Further research should extend the study across query distributions, languages, query categories, devices, and changing SERP presentations. The authors also identify limits from the sample, language and interface scope, and desktop-oriented observations.

  • Scope and sampling: The study is limited to 1,000 queries per search engine, split between 500 popular and 500 rare queries.The findings therefore provide evidence for those query types rather than the full distribution of search behavior.
  • Query variation: Future work should examine query categories, including commercial, multimedia, navigational, and transactional searches.The authors also propose studying query length and the middle of the frequency distribution, not only its extremes.
  • Language and interface scope: Results from one language and country-specific interface should not be broadly extrapolated to other languages or interfaces.The authors note that result-page arrangements can change across languages; for example, Ask.com does not show Ask 3D in European countries.
  • SERP monitoring: Future monitoring should track changes in presentation and host bias through organic rankings and shortcuts, using more sophisticated overlap analyses than top-10 URL comparisons.The authors expect sponsored links, shortcuts, and snippets to vary with query length and other conditions.
  • Device context: The findings describe normal computer screens, while mobile devices may force different trade-offs between organic and sponsored results.Small screens leave limited space for organic results and advertisements, so future SERP arrangements may differ.
  • User-visible results: Future studies should investigate which result combinations users see most often across query categories, lengths, screen sizes, and visible areas.The authors also suggest testing whether longer queries yield fewer paid listings and whether commercial queries produce more sophisticated results.
Loading 1511.05802v1…