PaperlayerView exact revision on arXiv

Source-linked AI summary

Search Arena: Analyzing Search-Augmented LLMs

Mihran Miroyan, Tsung-Han Wu, Logan King, Tianle Li, Jiayi Pan, Xinyan Hu, Wei-Lin Chiang, Anastasios N. Angelopoulos, Trevor Darrell, Narges Norouzi, Joseph E. Gonzalez

arXiv:2506.05334v2cs.CLcs.IRcs.LG

TL;DR

Existing search-augmented LLM evaluations are small and narrow, despite real-world interactions involving varied intents and user expectations beyond factual correctness. Search Arena gathers crowd-sourced multi-turn conversations and preference votes, finding that preferences track citation features and model performance varies across search and general-chat settings.

  • Problem

    Common evaluations of search-augmented LLMs are relatively small and focus on single-turn, monolingual, fact-based queries, while user expectations extend beyond factual correctness.

    Source: §1 · pp. 1–2

  • Method

    The authors crowd-sourced Search Arena, a human-preference dataset of in-the-wild interactions with search-augmented LLMs, including multi-turn conversations and paired judgments.

    Source: §1 · p. 2

  • Results

    Search-augmented models perform comparably in general chat while conventional LLMs underperform in search-intensive settings; users also prefer more cited sources, including responses with irrelevant citations.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    User preference can track citation presence even when citations do not support associated claims, and models behave differently across search and general-chat settings.

    Source: §1 · p. 2, §5 · p. 10

    The online human-in-the-loop evaluation restricts the scalability of testing new models.

    Source: §5 · p. 10

Abstract

from arXiv · show

Search-augmented language models combine web search with Large Language Models (LLMs) to improve response groundedness and freshness. However, analyzing these systems remains challenging: existing datasets are limited in scale and narrow in scope, often constrained to static, single-turn, fact-checking questions. In this work, we introduce Search Arena, a crowd-sourced, large-scale, human-preference dataset of over 24,000 paired multi-turn user interactions with search-augmented LLMs. The dataset spans diverse intents and languages, and contains full system traces with around 12,000 human preference votes. Our analysis reveals that user preferences are influenced by the number of citations, even when the cited content does not directly support the attributed claims, uncovering a gap between perceived and actual credibility. Furthermore, user preferences vary across cited sources, revealing that community-driven platforms are generally preferred and static encyclopedic sources are not always appropriate and reliable. To assess performance across different settings, we conduct cross-arena analyses by testing search-augmented LLMs in a general-purpose chat environment and conventional LLMs in search-intensive settings. We find that web search does not degrade and may even improve performance in non-search settings; however, the quality in search settings is significantly affected if solely relying on the model's parametric knowledge. We open-sourced the dataset to support future research. Our dataset and code are available at: https://github.com/lmarena/search-arena.

1 INTRODUCTION

Existing evaluations of search-augmented LLMs are limited in scale and scope, while user expectations extend beyond factual correctness. Search Arena provides a large human-preference dataset and analyzes response preferences and model performance across settings.

1 INTRODUCTION

  • Existing benchmarks such as SimpleQA and BrowseComp are relatively small and focus mainly on single-turn, monolingual, fact-based queries, unlike many real-world prompts requiring reasoning and open-ended dialogue.User expectations can also reflect citation number, relevance, credibility, and response presentation, not just factual correctness.Source: §1 · pp. 1–2
  • Search Arena released 24,069 conversations and 12,652 paired preference judgments from 11,650 users across 136 countries, 13 models, and around 70 languages.The dataset includes over 5,000 multi-turn interactions and introduces an intent taxonomy for search-enabled interactions.Source: §1 · p. 2
  • Users prefer responses with more cited sources and certain source types, while irrelevant citations also correlate positively with preference (β = 0.273).Correctly attributed citations have a positive association with preference (β = 0.285), raising concern that citation presence can influence perceived credibility without claim support.Source: §1 · p. 2
  • Conventional LLMs underperform in search-intensive settings (p-value = 0.009), while search-augmented models perform comparably in general chat, improving factual lookup (p-value = 0.012) with a slight text-processing decline (p-value = 0.077).The cross-arena results indicate that web search does not hurt and may improve performance across settings, while parametric knowledge alone is insufficient in search-intensive settings.Source: §1 · pp. 2–3

2 HUMAN PREFERENCE DATASET IN SEARCH

Search Arena collects crowd-sourced, multi-turn interactions with search-enabled models and documents their linguistic and intent diversity. Its prompts range from brief, under-specified queries to complex requests that differ from static factual benchmarks and general-chat prompts.

2 HUMAN PREFERENCE DATASET IN SEARCH

  • The open Search Arena platform collected over 24,000 conversations and 12,000 votes across 13 models, retaining conversation histories and system metadata such as reasoning traces and retrieved URLs.Users compared two anonymous model responses and could vote for their preferred response.Source: §2 · p. 3

2.1 LINGUISTIC AND CONVERSATIONAL DIVERSITY

  • Prompts span over 70 languages, with English at 58.3%, Russian at 11.8%, Chinese at 7.0%, and more than 11% multilingual; 22.4% of conversations are multi-turn.Multi-turn exchanges are typically clarifications or follow-up queries.Source: §2.1 · p. 3
  • Search Arena prompts average 57.1 words, between SimpleQA’s 16.3 and BrowseComp’s 103.3, and span brief, under-specified queries and longer, detailed requests.SimpleQA prompts are short and fact-oriented; BrowseComp prompts are intentionally long and constraint-heavy.Source: §2.1 · p. 4

2.2 INTENT DIVERSITY

  • An LLM-based differencing framework compared Search Arena prompts with SimpleQA, BrowseComp, and Text Arena to summarize their high-level prompt differences.GPT-4.1 generated candidate distinguishing properties and GPT-4.1-mini ranked them.Source: §2.2 · p. 4
  • Search Arena prompts emphasize analysis, synthesis, creative generation, and immediate functional assistance, unlike SimpleQA’s narrow fact retrieval and BrowseComp’s constrained investigative challenges.Compared with Text Arena, Search Arena focuses more on real-world factual lookup and decision support, while Text Arena prompts often seek problem-solving, programming, or creative help; Search Arena topics also include real-time subjects such as markets and health.Source: §2.2 · p. 4
  • A nine-category intent taxonomy was scaled with GPT-4.1 and validated with Cohen’s kappa of 0.812; factual lookup accounts for 19.3% of prompts, versus higher-order intents in the remainder.Factual lookup prompts average 17.2 words, compared with 66.7 words for the remaining prompts.Source: §2.2 · p. 4

3 PREFERENCE ANALYSES IN SEARCH

The section analyzes user preference votes across response features, citations, and search versus general-chat settings. Search augmentation does not hurt general-chat performance, while non-search models underperform in search settings; users also favor more citations, including irrelevant ones.

3 PREFERENCE ANALYSES IN SEARCH

  • The preference analysis uses normalized pairwise feature differences in a Bradley-Terry model, and its user votes are subjective rather than measures of objective correctness.Experts agreed with users on 68% of 100 samples when ties were excluded; the authors characterize the analysis as correlational, not causal.Source: §3 · pp. 4–5

3.1 GENERAL FEATURES

  • Reasoning models have over 60.0% average win rates among the top three models and analyze, rerank, and filter retrieved sources.The figure illustrates a reasoning trace with multi-document analysis, filtering, and synthesis.Source: §3.1 · p. 5, Figure 3 · p. 5
  • For sonar-pro, high search context yields a 63.9% average win rate versus 57.6% for medium context, while GPT-4o shows no significant difference.The analysis links larger search context to retrieving more web sources and more-preferred responses.Source: §3.1 · p. 5
  • Longer responses are positively associated with preference, but the response-length coefficient is 2.14 times smaller for Factual Lookup than for the full dataset.Factual Lookup responses average 168.3 words, compared with 422.8 for Creative Generation and 393.2 for Analysis.Source: §3.1 · pp. 5–6, Figure 5 · p. 7

3.2 CITATION FEATURES

  • Users prefer more citations, and supporting and irrelevant claim-citation pairs are both positively associated with preference.Contradicting pairs show no significant effect, so users do not distinguish supporting from irrelevant citations in this analysis.Source: Figure 6 · p. 7, §3.2 · p. 7

3.3 CROSS-SETTING ANALYSIS

  • Search augmentation performs comparably in general chat and improves factual lookup, while non-search models underperform in search settings.The search model was preferred for factual lookup and info synthesis in both arenas; text processing favored the non-search model in Text Arena.Source: §3.3 · pp. 8–9, Figure 8 · p. 8

4 RELATED WORK

Prior work spans traditional information retrieval, LLM-integrated retrieval, and benchmarks for search-augmented models. Search Arena addresses a gap in open-web search evaluation by providing human-preference data with full interaction traces across diverse, multi-turn prompts.

Large Language Models

  • LLM evaluation has moved from static benchmarks toward reasoning, coding, and open-ended dialogue, while crowd-sourced evaluations can still reflect biases in human preferences.The cited discussions include potential preference biases (Clark et al., 2021).Source: §4 · p. 9

4 RELATED WORK

  • Existing search benchmarks include SimpleQA and BrowseComp for single-turn fact-based queries, while WebArena emphasizes closed-world navigation rather than open-ended search and dialogue.Search Arena is introduced as a crowd-sourced dataset with human preferences, diverse intents, and multi-turn interactions across 70+ languages.Source: §4 · p. 9

Traditional and LLM-Integrated Information Retrieval (IR)

  • Traditional information retrieval research includes BM25, PageRank, and embedding-based approaches, alongside static benchmarks and large-scale web-search datasets with user logs and preference signals.The cited examples include BM25, PageRank, and web-search datasets (Craswell et al., 2020; Chen et al., 2024b).Source: §4 · p. 9
  • LLM-integrated retrieval includes RAG, where retrieved text is appended to the prompt, and benchmarks for retrieval, citation attribution, and question answering.Examples cited include RAG work and citation-attribution datasets.Source: §4 · p. 9
  • Search-augmented LLMs are framed as RAG variants with open-web search and multi-round query refinement, but existing benchmarks lack full human-AI interaction traces.Those traces include queries, retrieved documents, responses, and preferences.Source: §4 · p. 10

5 LIMITATIONS, CONCLUSION, AND FUTURE WORK

The section reports limitations of the crowd-sourced study and outlines future evaluation and model-development directions. It concludes with Search Arena’s dataset contribution and describes the appendix materials and platform.

5 LIMITATIONS, CONCLUSION, AND FUTURE WORK

  • Crowd-sourced results may reflect demographic skews, non-voting users, and subjective judgments, while releasing personal conversational data raises privacy and equitable-access concerns.The study anonymized responses, randomized their placement, obtained consent, and enforced a privacy policy.Source: §5 · p. 10
  • Future evaluation should compare human preferences with objective measures such as factuality, citation-claim consistency, and source specificity, because surface cues may not reflect content quality.The authors also propose offline benchmarks; the current online human-in-the-loop evaluation limits the scalability of testing new models.Source: §5 · p. 10
  • Future model work can improve source filtering and the backbone’s policy for when to search versus rely on parametric knowledge.The authors caution that blindly augmenting models with search is not always optimal.Source: §5 · p. 10
  • Search Arena comprises over 24,000 multi-turn conversations and 12,000 preference votes across more than 70 languages; preferences associate positively with citation count and certain source types, but citations are not always correct.The cross-arena experiment also finds that search-augmented and non-search models behave differently across settings; the dataset is released for future research.Source: §5 · p. 10

APPENDIX

  • The appendices cover platform design and data-collection policies, intent and topic methods, dataset differencing, model and preference analyses, citation analyses, cross-benchmark results, and related datasets.Appendix A also includes Terms of Service and daily-traffic figures.Source: APPENDIX · p. 17

A SEARCH ARENA PLATFORM

  • Search Arena’s interface presents two anonymous models side by side, with clickable citations and expandable references, and lets users vote during multi-turn conversations.The platform uses four feedback options: A is better, B is better, Tie, and Both are bad.Source: Figure A.1 · p. 17

A.1 DATA

The appendix describes supported-model configurations and citation-style controls, alongside figures for platform terms of service and daily traffic.

A.1 DATA

A.2 MODELS

  • Search Arena supports 12 search-augmented models from Perplexity, Gemini, and OpenAI, generally using provider APIs with default configurations and medium search context for Perplexity and OpenAI.Evaluated variants include OpenAI gpt-4o-search-preview with user location enabled and models configured with high search context.Source: §A.2 · p. 19
  • Citation styles are randomized between a standardized format and providers’ original styles to reduce model de-anonymization while enabling analysis of formatting effects.The inline citation style did not significantly affect user preferences or model rankings.Source: §A.2 · p. 19

B.1 LINGUISTIC FEATURES

The appendices characterize Search Arena’s users, prompts, intents, topics, and differences from other prompt datasets. They describe a geographically and linguistically diverse, multi-turn collection with varied prompt lengths and topic areas.

B.1 LINGUISTIC FEATURES

  • The prompts span 71 languages, with English at 56.4%, Russian at 11.4%, and Chinese at 6.8%; users come from 136 countries, led by the United States, Russia, Germany, and China.The dataset includes 11,650 unique users.Source: §B.1 · p. 19
  • Text Processing prompts average 85.1 words and Analysis prompts 72.7, compared with 24.7 for Explanation and 16.3 for Factual Lookup.Source: §B.1 · p. 19
  • 22.4% of conversations have more than one turn, including 3,288 two-turn, 966 three-turn, and 460 four-turn conversations.Source: §B.1 · p. 19

B.2 USER INTENT ANNOTATION

  • The intent taxonomy has nine categories and allows secondary labels for ambiguous or multi-purpose prompts; GPT-4.1 annotations achieved Cohen’s Kappa of 0.812 on top-two intents against human labels.Human annotator agreement ranged from 0.65 for primary labels to 0.79 for top-two matches.Source: §B.2 · p. 20

B.3 TOPIC MODELING

  • Topic modeling of 11,764 unique English prompts finds Technology Comparisons (22.0%), Market Analysis (12.3%), and Entertainment Characters (10.6%) among the most prevalent clusters, alongside a long tail of niche topics.Source: §B.3 · pp. 22–23, Figure B.4 · p. 23

C DESCRIBING DIFFERENCES

  • Prompt differences are analyzed with an LLM-based framework that generates candidate properties, filters and re-ranks them, then reports validation-set proportions for properties with p < 0.001.Source: C DESCRIBING DIFFERENCES · p. 25, §C.1 · p. 25

C.1 PROMPT DIFFERENCES

  • The cited comparisons report niche or lesser-known historical, scientific, or cultural topics at 51% vs 7%, recent or real-time information at 35% vs 7%, and requests for exact factual details at 97% vs 41%.Source: §C.1 · p. 25
  • The cited prompt categories report practical advice or real-world instructions at 29% vs 1%, and deductive-puzzle or investigative framing at 80% vs 3%, without identifying dataset assignments.Current factual information appears at 61% vs 25%, while analytical or step-by-step solutions appear at 21% vs 4%; the passages do not assign these percentages to datasets.Source: §C.1 · pp. 25–26

C.2 RESPONSE DIFFERENCES

The authors compare search and non-search model responses using LLM-proposed distinguishing properties, then report how often those properties appear in validation examples. The reported differences vary by prompt type and include content specificity, caveats, and formatting.

C.2 RESPONSE DIFFERENCES

  • 53.8% vs 41% of Factual Lookup responses included precise quantitative or technical specifics (p = 0.26).Source: §C.2 · p. 27
  • The reported property was incorporation of precise data, statistics, dates, names, and domain-specific terminology: 80.1% versus 61.9% (p = 0.18).For each dataset pair, the reported proportion is based on 100 validation-set examples, with corresponding p-values.Source: §C.2 · p. 27
  • 74.4% vs 56.4% of Text Processing responses used explicitly structured formatting (p = 0.1).The comparison is reported as non-search versus search model responses.Source: §C.2 · p. 27
  • Extensive background context or explanatory preambles appeared in 83.3% vs 72.2% of responses (p = 0.26).Source: §C.2 · p. 27
  • 71.7% vs 52.2% of Info Synthesis responses explicitly stated limitations, uncertainties, or caveats (p = 0.05).Source: §C.2 · p. 27

D.1 LEADERBOARD ANALYSIS

The section constructs Search Arena model ratings with Bradley-Terry regression rescaled to the Elo scale and compares leaderboard patterns across prompt subsets. The reported gaps differ between English and non-English prompts and between factual and non-factual prompts.

D.1 LEADERBOARD ANALYSIS

  • Model coefficients are calculated with Bradley-Terry regression and rescaled to match the Elo rating scale.The paper reports ratings and ranks for the full dataset and subsets by language and factuality.Source: §D.1 · p. 29
  • On non-English prompts, the gap between the two top models increases, with sonar-reasoning-pro-high performing better; Gemini-2.0-flash-grounding and Gemini-2.5-flash-grounding also perform better.Source: §D.1 · p. 29
  • The factual-prompt leaderboard shows a clear split, while the gap among the top three models, including sonar-pro-high, “zeros out” on non-factual prompts.Source: §D.1 · p. 30

D.2 RESPONSE LENGTH ANALYSIS

After controlling for response length, citation count, and citation sources, model scores and rankings tend to converge, especially within model families. Across the analyses, response length, citation counts, and cited-source distributions vary by model and prompt type.

D.2 RESPONSE LENGTH ANALYSIS

  • Reasoning models generally produce longer responses, and sonar-pro-high produces longer responses than sonar-pro with medium search context.Response lengths are uniform across OpenAI models; the response-length effect is statistically significant across intent categories but smallest for Factual Lookup prompts.Source: §D.2 · pp. 30–31, Figure D.7 · p. 32

E.1 CITATION COUNT

  • Models with larger search contexts cite more sources, while gemini-2.5-pro-grounding cites fewer than gemini-2.5-flash-grounding.The authors suggest the Pro model filters more sources from its final response.Source: §E.1 · p. 31
  • Citation-count effects are largest for Analysis prompts and not significant for Guidance prompts.Models with larger search contexts cite more sources, while reasoning models tend to cite fewer; the authors hypothesize that reasoning models filter irrelevant sources before final responses.Source: §E.1 · p. 31, Figure E.1 · p. 32, Figure D.6 · p. 32

E.2 CITATION SOURCES

  • Perplexity models tend toward social-media and community sources, OpenAI models toward mainstream news, and Gemini models lie between them.With the same search module, sonar-reasoning-pro-high cites more community and blog sources and fewer Wikipedia articles than sonar-pro-high.Source: Figure E.2 · p. 33, §E.2 · pp. 33–34, Figure E.3 · p. 34
  • Model scores and rankings tend to converge after controlling for response length, citation count, and citation sources, particularly among the top three Perplexity models.The authors note that factuality and coverage may correlate with citation counts or sources, and leave that analysis to future work.Source: §E.2 · p. 34

E.3 CITATION ATTRIBUTION ANALYSIS

The attribution analysis samples Search Arena conversations, retrieves cited web content, and labels claim–citation pairs as supporting, irrelevant, or contradicting. It aggregates those labels as Bradley-Terry model features and validates a subset with human experts.

E.3 CITATION ATTRIBUTION ANALYSIS

  • The analysis sampled around 100 conversations from each intent category, excluding the “Other” class, and collected over 20,000 web documents across 780 conversations.Samples with URLs that failed due to technical limitations, access restrictions, or licensing concerns were excluded.Source: §E.3 · p. 34
  • Gemini-2.5-flash-preview-0417 parsed messages into claim–URL pairs and judged each claim against scraped content as Support, Irrelevant, or Contradict.Source: §E.3 · p. 34
  • Counts of supporting, irrelevant, and contradicting claims per turn became Bradley-Terry features, and human experts validated a subset.Source: §E.3 · p. 35

F CROSS-SETTING ANALYSIS

Cross-benchmark performance and rankings differ by setting: ArenaHard-v2 aligns more closely with Search Arena than SimpleQA, but ranks reasoning models lower. The comparison with CORAL also highlights Search Arena’s broader, real-world preference-data scope.

F CROSS-SETTING ANALYSIS

  • The cross-setting evaluation compares BrowseComp and SimpleQA as search benchmarks with ArenaHard-v2 for non-search performance, using the same randomly sampled 500 questions per model.The shared subset was sampled once; model rankings are compared with Kendall’s tau and raw scores with Pearson correlation.Source: F CROSS-SETTING ANALYSIS · pp. 35–36
  • SimpleQA scores saturate at 89–93% (τ = 0.422, r = 0.582 agreement with Search Arena), while ArenaHard-v2 agrees more (τ = 0.556, r = 0.844) but ranks models differently, placing three reasoning models lower.ArenaHard-v2 agreement is higher than SimpleQA agreement; Search Arena (Other) agrees more with ArenaHard-v2 than Search Arena (Fact+Synth).Source: Table F.1 · p. 36, F CROSS-SETTING ANALYSIS · p. 36, Figure F.1 · p. 37

G.1 CORAL

  • Search Arena is a 24k real-world, preference-labeled dataset spanning 71 languages, unlike CORAL’s 8k English-only conversations with constructed answers over bounded document sets.Search Arena’s human preferences need not correlate with factuality or retrieval quality.Source: §G.1 · pp. 36–37
  • CORAL focuses on information-seeking queries and has longer conversations and more concise, less varied prompts than Search Arena, which spans broader intents and prompt lengths.Search Arena includes intents such as factual lookup and creative generation.Source: §G.1 · pp. 36–37

F CROSS-SETTING ANALYSIS

G.2 WILDCHAT

WildChat and Search Arena are both multilingual crowd-sourced conversation datasets, but differ in scale, tool access, and prompt length. Their turn-count distributions are similar, with Search Arena containing slightly more multi-turn conversations.

G.2 WILDCHAT

  • WildChat interactions use regular LLMs without tools such as web search, whereas Search Arena interactions use search-augmented LLMs.Source: §G.2 · p. 37
  • WildChat contains 4.8M conversations, compared with Search Arena’s 24k, and both datasets cover more than 70 languages.The comparison sampled 100k conversations from WildChat’s 3.2M non-toxic filtered version.Source: §G.2 · p. 38
  • WildChat and Search Arena have similar turn-count distributions, though Search Arena has slightly more multi-turn conversations; WildChat prompts average 594.9 words versus 57.1 in Search Arena.The prompt-length difference is attributed to differing user intents, including more real-time information-seeking queries in search-enabled settings.Source: §G.2 · p. 38, Figure G.2 · p. 38
Loading 2506.05334v2…