Source-linked AI summary
Towards Semi-Automatically Comparing Keyword-Based and Semantic Search Accuracy
Mohamed Ben Salha, Fiete Lüer, Maik Betka, Stefan Wagner
TL;DR
Direct quantitative comparison of keyword-based and semantic search remains lacking, despite their different output formats. The paper proposes a semi-automatic framework using query checklists, equivalence classes, and statistical tests; its industrial case study reports significant improvements in context-aware search over keyword methods. The framework is validated at one company, so its results are not necessarily generalizable to other industrial settings.
Problem
Objective comparison of semantic and conversational search with keyword systems remains lacking, despite the systems’ different output formats.
Source: §3 · p. 3Method
Results
The industrial case study found statistically significant improvements in context-aware search over keyword-based methods, with small and moderate advantages for RAG and its retrieval component, respectively.
Source: §7 · p. 7Takeaways & Limitations
The adaptable framework supports more objective evaluation of keyword and context-aware search and alleviates the need for costly, time-consuming subjective user studies.
Source: §7 · p. 7Validation involved only one company, so the results are not necessarily generalizable to other industrial environments and settings.
Source: §6.1 · p. 5
Abstract
from arXiv · showhide
The increasing importance of Information Retrieval (IR) in managing large datasets has highlighted significant limitations in traditional keyword-based search systems. Context-aware chat-based search methods, such as Retrieval Augmented Generation (RAG), have recently emerged, but their evaluation compared to keyword-based systems often relies on subjective user feedback. A rigorous, quantitative comparison between these paradigms remains lacking. This work introduces a novel, preliminary framework to quantitatively assess IR accuracy of search systems that produce different output formats, such as lists and messages. It focuses on two key aspects: the ranking accuracy for keyword-based systems and the completeness of retrieved information for semantic chat-based systems. Our approach enables semi-automatic comparisons of semantic and keyword-based methods using interchangeable equivalence classes tailored to domain-specific contexts (e.g., companies or problems). We validate the framework through an industrial case study, demonstrating statistically significant improvements in context-aware search over keyword-based methods, supported by analyses including the Mann-Whitney U-Test. With its adaptable design, the proposed framework provides a strong foundation for objectively assessing keyword-based and semantic chat-based search methods.
1 Introduction
Keyword search can miss context and typographical variants, while RAG incorporates retrieved context into generated answers. The paper proposes a preliminary quantitative framework and validates it in an industrial comparison of Confluence search and RAG.
1 Introduction
- RAG combines an external retrieval mechanism with an LLM’s text understanding and generation to provide retrieved context for answers.The introduction says this can support more context-aware and accurate responses and reduce hallucinations.Source: §1 · p. 1
- The paper addresses the missing direct quantitative comparison of RAG and keyword search by measuring information correctness and completeness.It proposes a preliminary framework for this evaluation.Source: §1 · p. 1
- The industrial case study compares Confluence’s keyword-based wiki search with RAG using interchangeable equivalence classes tailored to company- and problem-specific results.The authors also provide supplementary source code and resources for reproducing results.Source: §1 · p. 2
2 Goal
Keyword search returns ranked lists, whereas chat-based semantic systems produce text responses. Their different output formats make direct quality comparisons difficult, motivating an evaluation framework.
2 Goal
- Keyword-based systems return vertical lists, where higher-ranked entries are intended to be more relevant and users’ clicks decline down the page.The passage identifies Click-Through Rate (CTR) as the percentage of users who click a result after viewing it.Source: §2 · p. 2
- Chat-based semantic systems generally accept natural-language queries and produce text evaluated for completeness, hallucination, correctness, and faithfulness.Source: §2 · p. 2
- Figure 1 contrasts the different output formats, while the authors identify those differences as an obstacle to direct quality comparison.The figure passage also notes CTR for list-based entries.Source: §2 · p. 2, Figure 1 · p. 2
3 Related Work
Prior comparisons evaluated semantic and keyword search in list-based settings, while conversational search studies often used subjective user evaluations. Objective comparisons across differing output formats remain lacking.
3 Related Work
- Khan et al. [2014] evaluated five search engines on 10 queries, judging the first 20 documents for relevance and finding semantic web-based engines performed better.Their comparison covered systems returning lists, not chat-generated messages.Source: §3 · pp. 2–3
- Mesgar et al. [2019] evaluated a conversational literature-search agent using user assessments of diversity and information coverage; user studies can be costly, time-consuming, and biased by unrepresentative samples Almeida et al. [2017].Source: §3 · p. 3
- Objective evaluation of semantic or conversational systems against keyword search across different output formats remains lacking Zamani and Craswell [2020], Balog et al. [2020].Source: §3 · p. 3
4 Evaluation Framework
The framework compares IR approaches despite their different output formats by assessing correctness and ranking. It builds query checklists, assigns outputs to task-specific ordinal equivalence classes, and uses statistical tests to compare systems.
4 Evaluation Framework
- The framework quantitatively compares different IR output formats while accounting for result correctness and ranking.Source: §4 · p. 3
- The ground truth uses a checklist of expected content and queries covering exact keywords, synonyms, and synonym-based queries on unrelated topics.These query categories address direct term matching, semantic equivalence, and contextual reasoning across varied writing styles, lengths, and complexity.Source: §4 · p. 3
- Task-specific equivalence classes group outputs considered qualitatively similar and form an ordinal scale from complete retrieval to partial or no relevant information.Users define the classes and the results assigned to them.Source: §4 · p. 3
- Statistical tests, including the Mann-Whitney U-Test and Vargha-Delaney A12, assess differences between approaches and measure effect size.Source: §4 · p. 3
5 Experimental Setup and Results
The study tests the framework in an industrial case study, comparing Confluence with both the full RAG pipeline and its retrieval component. RAG performs better in both comparisons, and the reported differences are statistically significant.
5 Experimental Setup and Results
- The case study used Confluence and a German-language RAG application with hierarchical chunking and cosine-similarity retrieval at K=5.The 63 queries were created from interviewees’ example queries and varied in complexity, language, and length.Source: §5 · pp. 3–4
5.1 Confluence vs. RAG
- The industrial partner grouped outputs into three equivalence classes for all, some, or no expected information, with lower ordinal values indicating better accuracy.The classes were chosen with the case company and used click-through rate to reward salient results and penalize missing or less-salient information.Source: §5.1 · p. 4, Table 1 · p. 4
5.2 Confluence vs. RAG Retrieval Component
- RAG’s retrieval component was more accurate than Confluence, with Figure 3’s distributions corroborating an A12 statistic of 0.353.Confluence results were scored by rank through position 10; retrieval results were scored through position 5, with missing results assigned 10.Source: §5.2 · p. 4
5.3 Statistical Tests
- Both comparisons showed statistically significant differences at the p<0.05 threshold, assessed with the Mann-Whitney U-Test and Vargha-Delaney A12 statistic.Table 2 reports the test and effect-size results for 63 entries per group.Source: §5.3 · p. 4, Table 2 · p. 5
5.1 Confluence vs. RAG
- RAG achieved better IR performance than Confluence in 60% of evaluated cases, with A12=0.397.Figure 2 presents the aggregated comparison across 63 queries; higher equivalence-class values indicate worse performance.Source: §5.1 · p. 4, Figure 2 · p. 5
6 Discussion
The framework’s industrial validation demonstrates applicability to different IR systems, but the study identifies limits to its scope and opportunities for broader evaluation.
6 Discussion
- The framework is presented as adaptable to different IR systems, while the discussion identifies threats to validity and future-work directions.Source: §6 · p. 5
6.1 Threats to Validity
- Validation at one company does not establish generalizability to other industrial settings, and a manually created dataset may omit production-relevant entries or edge cases.Employees from the case company were involved to cover a broad spectrum of its underlying data.Source: §6.1 · p. 5
6.2 Future Work
- Future work could broaden the framework with a controlled, freely available ground truth, more automated ground-truth creation with manual checks, and metrics beyond accuracy.The authors also call for broader validation across demographic groups and industries and comparisons with more conversational search systems.Source: §6.2 · p. 6
7 Conclusion
The framework compares keyword-based and semantic search across different output formats using checklist-based evaluation and ordinal equivalence classes. An industrial case study reports statistically significant improvements for context-aware search over traditional keyword-based methods.
7 Conclusion
- The approach uses query datasets with checklists and ordinal-scaled equivalence classes to compare systems regardless of output format.Queries span language styles, lengths, and complexity; investigated criteria include correctness, completeness, and ranking.Source: §7 · pp. 6–7
- Context-aware search significantly improves over traditional keyword-based methods; RAG and its retrieval component show small and moderate advantages, respectively.The framework is validated in an industrial case study.Source: §7 · p. 7
- Statistical testing and domain-specific customization support more objective evaluation while alleviating the need for costly, time-consuming subjective user studies.The framework is preliminary and its investigated criteria are expandable.Source: §7 · pp. 6–7
8 GenAI Usage Disclosure
The disclosure describes AI-assisted coding and translation, with the authors reviewing and adjusting generated content.
8 GenAI Usage Disclosure
- ChatGPT generated statistical-test and plot snippets and the RAG chunking code; DeepL translated German resources, which the authors reviewed and adjusted for accuracy and clarity.The chunking method used the hierarchical structure.Source: §8 · p. 7