PaperlayerView exact revision on arXiv

Source-linked AI summary

C-SEO Bench: Does Conversational SEO Work?

Haritz Puerto, Martin Gubri, Tommaso Green, Seong Joon Oh, Sangdoo Yun

arXiv:2506.11097v3cs.CLcs.AIcs.IR

TL;DR

Existing C-SEO evaluations cover limited domains and single-adopter settings, leaving uncertain how methods perform across applications and competition. C-SEO Bench evaluates two tasks across six domains and varying adoption rates. The experiments find current C-SEO methods largely ineffective and often detrimental to ranking, while traditional SEO has greater impact.

  • Problem

    Existing C-SEO evaluations focus on limited applications and single-adopter settings, leaving their effectiveness across domains and competing adopters uncertain.

    Source: §1 · p. 2

  • Method

    C-SEO Bench evaluates citation-ranking changes for question answering and product recommendation across six domains, using an evaluation framework with varying adoption rates.

    Source: §1 · p. 2, §3 · p. 5

  • Results

    Current C-SEO methods are largely ineffective and can negatively affect document ranking, while traditional SEO has a significantly larger impact on citation rankings.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    Traditional SEO remains important in conversational search, while C-SEO gains decrease as more actors adopt the methods.

    Source: §1 · p. 2, §7 · p. 10

    The conclusions apply only to white-hat C-SEO; the paper does not evaluate black-hat methods or the interplay between traditional SEO and C-SEO.

    Source: §7 · p. 10

Abstract

from arXiv · show

Large Language Models (LLMs) are transforming search engines into Conversational Search Engines (CSE). Consequently, Search Engine Optimization (SEO) is being shifted into Conversational Search Engine Optimization (C-SEO). We are beginning to see dedicated C-SEO methods for modifying web documents to increase their visibility in CSE responses. However, they are often tested only for a limited breadth of application domains; we do not know whether certain C-SEO methods would be effective for a broad range of domains. Moreover, existing evaluations consider only a single-actor scenario where only one web document adopts a C-SEO method; in reality, multiple players are likely to competitively adopt the cutting-edge C-SEO techniques, drawing an analogy from the dynamics we have seen in SEO. We present C-SEO Bench, the first benchmark designed to evaluate C-SEO methods across multiple tasks, domains, and number of actors. We consider two search tasks, question answering and product recommendation, with three domains each. We also formalize a new evaluation protocol with varying adoption rates among involved actors. Our experiments reveal that most current C-SEO methods are not only largely ineffective but also frequently have a negative impact on document ranking, which is opposite to what is expected. Instead, traditional SEO strategies, those aiming to improve the ranking of the source in the LLM context, are significantly more effective. We also observe that as we increase the number of C-SEO adopters, the overall gains decrease, depicting a congested and zero-sum nature of the problem. Our code and data are available at https://github.com/parameterlab/c-seo-bench and https://huggingface.co/datasets/parameterlab/c-seo-bench.

1 Introduction

Existing C-SEO evaluations cover too few application domains and adopters to show whether methods generalize or remain effective under competition. C-SEO Bench broadens evaluation across tasks, domains, and actors, finding current methods mostly ineffective while traditional SEO remains more influential.

1 Introduction

  • Earlier benchmarks focused on a single application and one adopting actor, despite C-SEO methods potentially behaving differently across applications and domains.The paper motivates coverage of multiple tasks, domains, and actors to evaluate domain-specific effects.Source: §1 · p. 2
  • C-SEO Bench evaluates methods across web-question answering and product recommendation, with three domains for each task and varying numbers of adopting actors.It measures effectiveness by whether a modified document is cited earlier than before.Source: §1 · pp. 2–3
  • The authors frame their analysis as challenging assumptions about C-SEO, finding that current methods largely remain ineffective.They conclude C-SEO will complement rather than replace SEO.Source: §1 · p. 2
  • Traditional SEO’s retrieval-ranking improvements have a far larger effect on document importance in LLM responses than current C-SEO methods, which can even lower document ranking.The comparison spans all six domains; C-SEO generally yields close to zero ranking boost, and its marginal gains shrink as adoption increases.Source: §1 · pp. 2–3, Figure 1 · p. 2

2 Background and Related Work

Prior work addresses citation quality and ways to influence citation ranking, but its datasets and evaluation settings leave gaps in coverage and realism. This paper situates its focus on benign content improvements rather than attacks on LLM robustness.

Influencing Citation Ranking

  • Citation-generation benchmarks assess whether LLM responses are fluent, correct, and faithful in their citations, while ranking-influence work seeks to change which documents are cited earlier.Early systems were trained to cite source documents [Nakano et al., 2021, Menick et al., 2022], and Gao et al. [2023] proposed a benchmark for grounding and citation quality.Source: §2 · p. 3

Black-hat C-SEO

  • Black-hat C-SEO studies use adversarial methods, but most rely on small datasets or narrow domains, limiting conclusions to attack feasibility rather than broad performance.The cited studies include 50 pages of fictitious products and news, ten fictitious coffee machines, and a 1k-product dataset limited to five categories.Source: §2 · p. 3

White-hat C-SEO

  • Aggarwal et al. [2024] evaluate white-hat stylistic changes using word count, but their benchmark covers one task, lacks diverse domains, and assumes a single adopter.Word count does not capture improvements in tasks such as product recommendation.Source: §2 · p. 3
  • Bardas et al. [2025] assume the user query is known beforehand, limiting applicability where real-world queries are unknown; this paper instead evaluates content improvement, not LLM robustness.The authors use white-hat C-SEO methods for that scope.Source: §2 · p. 4

3 Problem Definition

The paper models conversational search as an LLM generating a response conditioned on a query and retrieved documents, then defines C-SEO by the goal of improving a document’s citation position. It distinguishes retrieval-focused SEO from document-focused C-SEO and formalizes competition through adoption rates.

3 Problem Definition

  • A conversational search engine conditions LLM output on a query and external documents retrieved by a search engine, producing inline citations to those sources.The output is sampled from the query and retrieved-document context.Source: §3 · p. 4
  • C-SEO modifies a document to improve its position in the LLM’s ordered citations, equivalently maximizing its learned relevance score or minimizing its citation rank.The best case is being cited first.Source: §3 · p. 4
  • SEO targets a document’s retrieval rank, whereas C-SEO changes the document to make it more appealing to the LLM.SEO is expected to affect citation ranking indirectly through retrieval position.Source: §3 · pp. 4–5
  • The multi-adopter setup models content providers as noncooperative players choosing C-SEO methods to improve citation-rank utility; adoption rate is the share of players using a method.Unilateral adoption is the special case where one player adopts while others remain static, as in most prior work [Kumar and Lakkaraju, 2024, Tang et al., 2025, Pfrommer et al., 2024, Aggarwal et al., 2024].Source: §3 · p. 5

4 C-SEO Bench

C-SEO Bench evaluates methods across product recommendation and question answering, with multiple domains in each task. Its evaluation varies how many retrieved documents adopt a method and measures gains across adoption rates.

4 C-SEO Bench

  • Each task spans multiple domains to represent varied real-world use cases and enable analysis of domain-specific effects.Question answering includes web questions, news, and debate; the benchmark’s datasets also include retail, video games, and books.Source: §4 · p. 5, §4.1 · p. 6

4.1 Datasets Construction

  • The benchmark evaluates six datasets across question answering and product recommendation, with more than 1.9k queries and 16k documents.The six domains support assessment of both generalization and domain-specific effects.Source: §4 · p. 5, §4.1 · p. 6

4.2 Evaluation

  • At adoption rate α, the protocol measures citation-rank improvement against the no-method baseline and tests whether rank decreases significantly.It uses a right-tailed Wilcoxon signed-rank test with Holm-Bonferroni-adjusted p-values.Source: §4.2 · p. 6
  • AUC summarizes overall gains across adoption rates from 0% to 100%, with the trapezoidal rule approximating the metric for discrete adoption rates.Higher AUC indicates higher average gains across adoption rates, relevant when the adoption rate is unknown.Source: §4.2 · p. 6

5 C-SEO Methods

The benchmark tests ten content-transformation methods intended to improve document citation ranking. These include eight transformations, a combined improvement method, and a prompt-generated summary appended to the document.

C-SEO Methods

  • The remaining transformations add technical terms, simplify language, or add quotations to increase credibility, readability, or trust.The eight methods also include fluency and unique-word changes, as well as authority, statistics, and citations.Source: §5 · p. 7
  • Content Improvement combines eight transformations into one holistic text-improvement method.The transformations aim to increase authority, perceived technical depth, credibility, fluency, uniqueness, readability, or trust.Source: §5 · p. 7
  • LLM Guidance generates a markdown summary and concatenates it at the beginning of the original document to guide the LLM about its content.The method is inspired by the LLMs.txt standard.Source: §5 · p. 7
  • The benchmark evaluates ten methods by transforming retrieved documents, then asking an LLM to answer queries and cite those documents.The transformation and conversational search runs use gpt-4o-mini-2024-07-18; the CSEs also include claude-3-5-haiku-20241022, o3, and o4-mini.Source: §5 · p. 7

6 Experiments

The experiments test C-SEO effectiveness, traditional SEO, and adoption-rate effects across multiple tasks, domains, and models. C-SEO methods usually produce little or no significant gain and can lower rankings, while document position in the LLM context has a stronger effect; gains shrink as adoption grows.

6 Experiments

  • The evaluation asks whether C-SEO improves citation rankings, whether traditional SEO remains impactful, and how adoption rates affect C-SEO.Unilateral-adoption tests randomly select one document; adoption-rate experiments use at most ten actors, so 10% represents one actor and 100% represents ten.Source: §6 · p. 7, §6.1 · p. 8

6.1 Experimental Setup

  • The experiments run C-SEO methods with gpt-4o-mini and evaluate conversational search using gpt-4o-mini, Haiku 3.5, o3, and o4-mini.The authors use default provider decoding parameters.Source: §6.1 · p. 8

6.2 Main Results

  • 61.0% of retail product ranks remain unchanged after LLM guidance, while its positive and negative shifts partly cancel, leaving the overall average effect close to zero with high variance.Retail guidance produces a positive boost in 26.2% of cases and a negative boost in 12.8%.Source: §6.2 · pp. 8–9
  • Most C-SEO methods show limited effectiveness, and many significantly reduce document rankings rather than improving them.Only three of 54 cases show statistically significant ranking improvements; no method is effective for question answering or for Haiku 3.5.Source: §6.2 · p. 9
  • 26 out of 30 product-recommendation cases on Haiku 3.5 and 19 out of 30 question-answering cases on o4-mini show significant negative effects.The Statistics method decreases rankings in 19 out of 24 evaluated settings.Source: §6.2 · p. 9
  • The authors explain that earlier optimism about C-SEO does not conflict with their results because word count does not measure citation preference, and position-adjusted word-count results also generally decrease.Their evaluation uses citation ranking as the measure of LLM preference.Source: §6.2 · p. 9

6.3 Importance of Traditional SEO

  • Placing a document first in the LLM context yields far greater citation-ranking gains than any C-SEO method, and the top three positions produce the largest boosts.At least the first two positions yield significant boosts across all domains; positions 8, 9, and 10 can degrade performance when they move a document below its original position.Source: §6.3 · p. 9, Figure 4 · p. 10

6.4 C-SEO Behaves as a Zero-Sum Game

  • C-SEO gains decline steadily as more players adopt a method, with benefits eventually converging toward zero at full adoption.The authors describe this as a congested, zero-sum game: gains for one citation's rank come at the expense of others.Source: §6.4 · p. 10, Figure 5 · p. 10

7 Conclusion

C-SEO Bench spans two tasks and six domains; its experiments find current C-SEO methods largely ineffective and sometimes harmful to document ranking, while traditional SEO has a larger impact on citation rankings. The work is limited to white-hat C-SEO and notes potential dual-use of the benchmark for black-hat attacks.

7 Conclusion

  • Across two tasks and six domains, current C-SEO methods are largely ineffective and can lower document ranking, while traditional SEO has a significantly larger impact on citation rankings.Source: §7 · p. 10
  • The game-theoretic analysis characterizes C-SEO as a congested, zero-sum game, with incentives to keep investing in methods and potential risks from current techniques.Source: §7 · p. 10
  • The findings cover white-hat methods only, not black-hat methods, malicious manipulation, interplay with traditional SEO, non-proprietary models, documents exceeding context limits, or non-English content.Source: §7 · p. 10

A Broader Impacts

  • The benchmark is not designed for black-hat methods but could potentially test attacks that downgrade competitors’ citation rankings, creating dual-use concerns.Source: A Broader Impacts · p. 13

B License and Terms of Use of Used Assets

The listed datasets permit research use, and the authors state that they comply with their terms. The datasets have licenses ranging from standard licenses to a custom noncommercial and research-use license.

B License and Terms of Use of Used Assets

C Author Contributions

All authors contributed to experiment design, analysis, iteration, writing, and editing. Individual contributions included data and experiment work, supervision and statistical support, technical support, and organizational and funding contributions.

C Author Contributions

  • All authors contributed to experiment design, analysis, iteration, writing, and editing; Tommaso Green and Martin Gubri also provided technical support.Source: C Author Contributions · p. 13
  • Haritz Puerto collected and cleaned data, implemented experiments, drafted the paper, reviewed related work, analyzed results, prepared visualizations, and published research artefacts.Source: C Author Contributions · p. 13
  • Martin Gubri helped shape the narrative, proposed Sections 6.4 and Figure 4, created Figure 2, contributed to the draft, and supported statistical testing.Source: C Author Contributions · p. 13
  • Puerto, Green, and Gubri proposed the LLM guidance and content improvement methods; Oh and Yun contributed organizational and funding support, with Oh suggesting Figure 1 and Yun proposing Section 6.3.Source: C Author Contributions · p. 13
  • Gubri, Oh, and Yun provided weekly supervision, while Green provided weekly feedback.Source: C Author Contributions · p. 13

D Other Results

Supplementary results report that LLM Document Optimization methods do not achieve significant gains on o3 or o4-mini, whereas Best SEO is always statistically significant. Other tables report SEO rank improvements by context position and complementary p-values across models and domains.

D Other Results

  • Table 5 reports average and standard deviation of gpt-4o-mini SEO rank improvements by document position in the LLM context.Source: Table 5 · p. 14
  • Complementary p-value tables cover SEO baselines and C-SEO methods across gpt-4o-mini, Haiku 3.5, o3, and o4-mini.Source: Table 6 · p. 14, Table 7 · p. 15, Table 8 · p. 15
  • On o3 and o4-mini, no LLM Document Optimization method achieves significant gains, while Best SEO is always statistically significant.Source: Table 9 · p. 15, Table 11 · p. 16

E Prompts for C-SEO Methods

The appendix presents prompts for C-SEO methods and dataset construction. It describes an llms.txt template and content-rewriting prompt, alongside prompts for generating book, web, and news queries.

E Prompts for C-SEO Methods

  • The appendix says methods from Aggarwal et al. [2024] use the original paper’s prompt, while prompts for the authors’ methods are shown here.Source: E Prompts for C-SEO Methods · p. 17
  • The llms.txt prompt creates a Markdown file that summarizes the main text and provides brief background, guidance, and links when available.Source: E Prompts for C-SEO Methods · p. 17
  • The llms.txt template uses a title, named sections with details, and the supplied text as input.Source: E Prompts for C-SEO Methods · p. 17
  • The content-improvement prompt asks for fluent, authoritative, persuasive, assertive prose and clearer structure while preserving the source’s meaning and information.Source: E Prompts for C-SEO Methods · p. 17

F Prompts for Dataset Construction

Query Expansion for Web Dataset

Query Generation for News

Related papers

Loading 2506.11097v3…