PaperlayerView exact revision on arXiv

Source-linked AI summary

The Retrieval Effectiveness of Web Search Engines: Considering Results Descriptions

Dirk Lewandowski

arXiv:1511.05800v1cs.IR

TL;DR

The study addresses whether search engines differ in result quality and whether descriptions, often underrepresented in earlier evaluations, should inform relevance assessment. It compares engines using judgments of results and descriptions, finding no significant overall difference between Google and Yahoo, while Google provides more relevant descriptions but also has more description deception than most competitors.

  • Problem

    Earlier retrieval studies often underweighted results descriptions, despite uncertainty about whether Google’s dominance reflects better results or user familiarity.

    Source: Introduction · p. 2

  • Method

    The study compares search engines’ results and descriptions, considering whether descriptions predict relevant results and how paired judgments affect relevance assessment.

    Source: Results · pp. 14, 16

  • Results

    Google and Yahoo show no significant overall difference; Google has a larger ratio of relevant descriptions but also more description deception than other engines except MSN.

    Source: Conclusion and Future Work · pp. 19–20

  • Takeaways & Limitations

    Search engines should improve result precision while also making descriptions relevant and consistent with the results they describe.

    Source: Conclusion and Future Work · p. 20

    The study models users as systematically evaluating a fixed amount of results, which may not reflect real search behavior.

    Source: Conclusion and Future Work · p. 20

Abstract

from arXiv · show

Purpose: To compare five major Web search engines (Google, Yahoo, MSN, Ask.com, and Seekport) for their retrieval effectiveness, taking into account not only the results but also the results descriptions. Design/Methodology/Approach: The study uses real-life queries. Results are made anonymous and are randomised. Results are judged by the persons posing the original queries. Findings: The two major search engines, Google and Yahoo, perform best, and there are no significant differences between them. Google delivers significantly more relevant result descriptions than any other search engine. This could be one reason for users perceiving this engine as superior. Research Limitations: The study is based on a user model where the user takes into account a certain amount of results rather systematically. This may not be the case in real life. Practical Implications: Implies that search engines should focus on relevant descriptions. Searchers are advised to use other search engines in addition to Google. Originality/Value: This is the first major study comparing results and descriptions systematically and proposes new retrieval measures to take into account results descriptions

Introduction

Search-engine quality spans several dimensions, and this study focuses on result quality. It asks whether engines differ in retrieval effectiveness and whether results descriptions should be included in judging that effectiveness.

Introduction

  • Search-engine quality includes index quality, result quality, search features, and usability, while this study examines result quality alone.The authors hope future work will develop a balanced model that includes all four areas.Source: Introduction · p. 1
  • Google’s European market dominance coincides with a knowledge gap about whether it provides better results than comparable alternatives.The authors contrast genuinely better results with user familiarity as possible explanations for Google’s market position.Source: Introduction · p. 2
  • Prior retrieval-effectiveness studies may not reflect the changed search-engine landscape and often underweight the importance of results descriptions.The authors identify these as two reasons for conducting a new comparison.Source: Introduction · p. 2
  • The study asks which engines perform best, whether descriptions accurately indicate relevance, and how descriptions affect relevance measures and overall judgments.Its questions also ask whether engine performance has changed since earlier studies.Source: Introduction · p. 2

Literature Overview

Earlier search-engine evaluations varied in methods and generally assessed retrieved documents rather than the descriptions that help users decide what to open. The review highlights limited prior work comparing descriptions with results directly.

Literature Overview

  • Prior studies used varied query sets, jurors, relevance judgments, and result cutoffs, with most considering only the first 10 or 20 results.The review also notes that lists were commonly anonymized, while randomization was uncommon in the studies it examined.Source: Literature Overview · pp. 3–4
  • A 2004 study found Google’s precision at 20 results was 0.59, compared with 0.53 for Lycos and 0.51 for AltaVista, while overall precision remained low.The review also notes that its coverage of retrieval-effectiveness studies is selective rather than exhaustive.Source: Literature Overview · pp. 2, 4
  • Earlier evaluations often found weak retrieval performance; one French-language study rated Yahoo and Google highest at 2.3, followed by MSN at 2.0.The study reported that no tested engine achieved its passing grade of 2.5.Source: Literature Overview · p. 4
  • Search-result descriptions matter because users use them to predict document relevance before opening documents, but conventional evaluations typically judge only the documents.Predictive judgments are part of the search process and are omitted from those conventional evaluations.Source: Literature Overview · pp. 4–5
  • Search engines generate descriptions from several sources, including page text, directory descriptions, and page metadata, each with coverage or interpretability trade-offs.Keyword-in-context snippets can be incomplete; directory and metadata descriptions are unavailable for many pages.Source: Literature Overview · p. 5
  • Only two studies considered results descriptions in performance evaluation: Wolff used descriptions alone, while Griesbaum compared them with results and found differing judgments for about one third of hits.Griesbaum considered the findings tentative because different jurors judged descriptions and results.Source: Literature Overview · p. 6

Methods

The study compares five engines using German informational queries from participants’ everyday searches. It collects anonymous, randomized judgments of both result descriptions and the underlying results.

Methods

  • The study compares Google, Yahoo, MSN, Ask.com, and Seekport, selecting engines that provide their own web indexes and are major or notable services.Seekport was included as a notable German engine with its own index.Source: Methods · pp. 6–7
  • Forty student participants supplied recent private-use queries in German and described the underlying information need.Each participant later judged results for the query they had originally submitted.Source: Methods · pp. 7–8
  • The evaluation is limited to informational queries, so its conclusions do not generalize to all search-engine queries.The authors argue that query types should be evaluated separately because their relevance judgments should differ.Source: Methods · p. 7
  • The study used 40 queries because of participant availability and judgment workload, a count within the cited recommended range of 25 to 50.The queries averaged 2.3 terms and differed from general German query distributions, including overrepresentation of education and society and underrepresentation of commerce.Source: Methods · pp. 7–8
  • For each query, the researchers examined the first 20 results from each engine to enable comparison with earlier studies and with description-result pairs.The cutoff was also motivated by user studies indicating that searchers tend to ignore results beyond the first few.Source: Methods · p. 9
  • Participants made binary judgments of results and descriptions, first judging anonymized, randomized descriptions and then judging the linked results without seeing the descriptions.A description counted as relevant when the participant thought it pointed to a relevant result.Source: Methods · pp. 9–10

Results

Yahoo and Google lead on relevant results, with no significant difference between them, while Google leads on relevant result descriptions. Rankings vary by cutoff and query, and description-result measures reveal additional mismatches.

Results

  • 48.5% of Yahoo’s results and 47.9% of Google’s were relevant, a nonsignificant difference; relevant descriptions were 60.2% for Google and 52.8% for Yahoo, with significant differences across engines.The other three engines had relevant-description ratios below 50%.Source: Results · p. 10, Figure 2 · p. 11
  • Google and Yahoo returned at least one relevant result for every query; Ask.com missed one query, while MSN and Seekport each missed three.Source: Results · p. 11
  • Yahoo answered 16 queries best across the complete 20-result sets, but no engine led every query; Google and Yahoo most often ranked first through third.The authors therefore give no clear recommendation for a single engine, noting performance depends largely on the query.Source: Results · p. 12, Figure 4 · p. 13
  • For the top three results, Google and Yahoo answered more than 20 queries best, while Google and Yahoo also performed best at the top ranks overall.Ask.com’s full-set performance was relatively good, but its top-rank results lagged, suggesting ranking optimization was needed.Source: Results · pp. 11, 13, Figure 3 · p. 12, Figure 5 · p. 13
  • DRdist was 0.12 for Google and 0.11 for MSN, versus 0.05 for Yahoo and 0.07 for both Ask.com and Seekport; descriptions generally had higher precision than results.The precision-distance values varied little by result position.Source: Results · p. 14, Figure 7 · p. 16
  • On the top 20, Google and Yahoo remained best in description-result precision without significant differences; Google outperformed all others at first position, while engines adjusted as more results were counted.The top-20 difference between Google and Ask.com was significant only at the 0.05 level.Source: Results · pp. 17–18, Figure 8 · p. 18
  • 0.79 was Seekport’s DRconf, above the other engines’ 0.70–0.73, although this measure also rewards jointly irrelevant descriptions and results.Overall, descriptions allowed users to judge relevance in about 70% of cases, but were misleading in about 30%.Source: Results · p. 18
  • Description fallout was relatively low for all engines, but Ddec reached 0.21 for Google and 0.20 for MSN, meaning relevant-looking descriptions pointed to irrelevant results in those proportions of cases.Source: Results · p. 19

Conclusion and Future Work

Google and Yahoo perform similarly overall, but Google provides more relevant result descriptions, which may partly explain its stronger perceived performance. The conclusions are bounded by a systematic-results user model, and the authors propose adapting future tests to changing result presentations.

Conclusion and Future Work

  • Google and Yahoo were not significantly different overall; Google led in relevant descriptions, while their precision differences disappeared beyond seven results.Google led in precision through the seventh result, but differences were insignificant when considering more than seven results.Source: Conclusion and Future Work · p. 19
  • Google's large ratio of relevant-looking descriptions may partly explain its stronger interview performance, though users often click only one or a few results.The authors assume relevant-looking descriptions contribute in part to Google's good performance in user interviews.Source: Conclusion and Future Work · p. 20
  • Google's many relevant descriptions come with a high description-deception value, pointing users to irrelevant results more often than other engines except MSN.Source: Conclusion and Future Work · p. 20
  • Ask.com outperformed Seekport and MSN, while MSN generally disappointed.The authors note that Ask.com's relatively good performance may be surprising given its recent start in Germany.Source: Conclusion and Future Work · p. 20
  • At 20 results, precision ranges from 0.37 to 0.52, while description-result precision ranges from 0.27 to 0.41.Source: Conclusion and Future Work · p. 20
  • The authors recommend improving result precision while also improving description relevance and description-result conformance.Source: Conclusion and Future Work · p. 20
  • The study evaluates a systematic amount of results and their descriptions, a user model that may not reflect real behavior.Many users instead click until they find one or a few suitable results, then stop searching.Source: Conclusion and Future Work · p. 20
  • Future tests will adapt to changing result presentations; the current study is presented as a first step toward user-aware retrieval measures.Source: Conclusion and Future Work · p. 20

Notes

The notes clarify which results were evaluated and how ties were ranked. They also explain the study's continued use of the familiar MSN name.

Notes

  • The study used only organic results, excluding sponsored listings and special results such as news and weather.Source: Notes · p. 20
  • Microsoft's search engine had changed its name to Live.com, but the study uses MSN because users knew that name and it remained the portal's name.Source: Notes · p. 20
  • Engines tied on a query shared a rank, and the next rank was skipped.For example, if the first and third engines tie at rank 1, the second engine receives rank 3.Source: Notes · p. 20
Loading 1511.05800v1…