PaperlayerView exact revision on arXiv

Source-linked AI summary

USimAgent: Large Language Models for Simulating Search Users

Erhan Zhang, Xingzhu Wang, Peiyuan Gong, Yankai Lin, Jiaxin Mao

arXiv:2403.09142v2cs.IRcs.AI

TL;DR

Accurately simulating search users is difficult because search actions depend on complex cognitive processes and interdependent behaviors. USimAgent uses LLM-guided reasoning, querying, clicking, and stopping to generate complete sessions, outperforming existing methods on query generation and matching traditional methods on click and stopping prediction.

  • Problem

    Existing user simulators struggle to reproduce complex, interdependent search behavior driven by users’ information needs, knowledge, learning, and reasoning.

    Source: §1 · p. 1

  • Method

    USimAgent uses LLMs to generate reasoning, stopping decisions, query reformulations, clicks, and observations across context-aware search rounds.

    Source: §1 · p. 2, §3 · p. 2

  • Results

    USimAgent outperforms existing methods in query generation and is comparable to traditional methods for predicting clicks and stopping behaviors.

    Source: §1 · p. 2, §5 · p. 4, §4.2.1 · p. 4

  • Takeaways & Limitations

    The results support using LLMs for user simulation and motivate developing more robust and generic user simulators.

    Source: §1 · p. 2, §5 · p. 4

    USimAgent’s predictive accuracy may still fall short of models trained on more extensive datasets.

    Source: §5 · p. 4

Abstract

from arXiv · show

Due to the advantages in the cost-efficiency and reproducibility, user simulation has become a promising solution to the user-centric evaluation of information retrieval systems. Nonetheless, accurately simulating user search behaviors has long been a challenge, because users' actions in search are highly complex and driven by intricate cognitive processes such as learning, reasoning, and planning. Recently, Large Language Models (LLMs) have demonstrated remarked potential in simulating human-level intelligence and have been used in building autonomous agents for various tasks. However, the potential of using LLMs in simulating search behaviors has not yet been fully explored. In this paper, we introduce a LLM-based user search behavior simulator, USimAgent. The proposed simulator can simulate users' querying, clicking, and stopping behaviors during search, and thus, is capable of generating complete search sessions for specific search tasks. Empirical investigation on a real user behavior dataset shows that the proposed simulator outperforms existing methods in query generation and is comparable to traditional methods in predicting user clicks and stopping behaviors. These results not only validate the effectiveness of using LLMs for user simulation but also shed light on the development of a more robust and generic user simulators. The code and data are accessible at https://github.com/Meow-E/USimAgent.

1 INTRODUCTION

USimAgent addresses the difficulty of reproducing complex, interdependent search behavior by using LLMs to simulate complete search sessions. It generates queries, clicks, and stopping actions, outperforming existing methods for query generation while matching traditional methods on clicks and stopping.

  • Traditional simulators model queries, clicks, and stopping as separate steps, overlooking how information needs, background knowledge, learning, and reasoning jointly drive search behavior.Source: §1 · p. 1
  • LLMs offer language understanding, zero-shot/few-shot adaptation, multitask capability, and reasoning for simulating diverse, coherent user interactions.Source: §1 · pp. 1–2
  • USimAgent uses LLMs to alternate query and click generation until stopping, while integrating session context and external information without task-specific training.Its reasoning-and-action design supports coherent behavior across a complete interaction sequence.Source: §1 · p. 2
  • On a public user behavior dataset, USimAgent outperforms existing methods in query generation and is comparable to traditional models for click and stopping behaviors.The evaluation compares generated interactions with real user behavior data.Source: §1 · p. 2

2 RELATED WORK

Prior user simulators separately model query generation, clicking, and stopping, while LLM-agent research has only partially extended simulation to contextual query generation. USimAgent targets the broader session-level problem by combining these capabilities with reasoning.

  • Query simulators commonly sample terms from document- or topic-associated language models, whereas click models use probabilistic or neural interaction models.Source: §2 · p. 2
  • Stopping strategies mainly rely on heuristic judgment rules, leaving traditional approaches fragmented across search behaviors.Source: §2 · p. 2
  • Existing LLM-based efforts generate contextual or reformulated queries but remain largely limited to single-query simulation rather than coherent search sessions.Source: §2 · p. 2
  • USimAgent extends LLM multitasking and reasoning from query generation to broader session-level user behavior simulation.Source: §2 · p. 2

3 METHODOLOGY

USimAgent generates complete search interaction sequences through repeated LLM-guided reasoning, stopping decisions, query reformulation, and click behavior. Each round updates the context with observations from clicked results before the next decision.

Problem formulation

  • Given task T, USimAgent generates an interaction sequence of alternating queries and clicks that ends with a stop action.Source: §3 · p. 2

Reasoning before acting

  • At each round, the agent reasons from prior context, decides whether to stop, and otherwise proceeds to query reformulation.Source: §3 · p. 2

Query reformulation

  • Query reformulation uses an LLM prompt conditioned on the task and current context, then appends the generated query to both context and interaction history.Source: §3 · p. 2

Click Prediction

  • For click prediction, the agent receives SERP titles and abstracts and selects results according to task relevance.Source: §3 · p. 2

Overall framework

  • The framework repeats reasoning and action, query generation, and click behavior across rounds while using context from previous interactions.Source: Figure 1 · p. 3

Click Prediction

  • After clicking, the agent reads detailed result content to generate observations, updates the context, and begins another reasoning round.Source: §3 · p. 3

4 EXPERIMENTS

Experiments evaluate USimAgent on a filtered real-user search dataset against query, click, and stopping baselines, finding strong query-generation performance and mixed results for clicks and stopping. Ablations show that session context is especially important for query generation and stopping decisions.

4.1.1 Datasets.

  • USimAgent was evaluated on 164 complete sessions from 40 users after excluding invalid, incomplete, and unsuitable task data.The dataset retained only the top ten SERP results when lists exceeded ten.Source: §4.1.1 · p. 3

4.1.2 Baselines and Evaluation Metrics.

  • The experiments compare USimAgent with probabilistic query models, traditional and neural click models, and heuristic stopping strategies using BLEU, accuracy, precision, recall, and F1.PBM was additionally trained with a regression-based EM algorithm to address data sparsity.Source: §4.1.2 · p. 3

4.2.1 Main Results.

  • USimAgent achieved the best similarity to real queries among the evaluated query-reformulation methods.Popular Selection outperformed Random Selection because it accounts for word importance, while USimAgent achieved the best result.Source: §4.2.1 · p. 3, Table 1 · p. 4
  • USimAgent’s click-prediction performance was comparable to traditional models, while UBM performed better and NCM performed poorly, likely because of sparse-data underfitting.The authors attribute USimAgent’s weaker performance relative to classic click models partly to limited position-bias modeling and zero-shot prediction.Source: §4.2.1 · p. 4, Table 2 · p. 4
  • For stopping prediction, USimAgent outperformed baselines in Accuracy and Precision but not in Recall or F1 Score.The comparison used the stopping-strategy baselines listed for the experiment.Source: §4.2.1 · p. 4, Table 3 · p. 4

4.2.2 Ablation Study.

  • Removing context substantially harmed query generation and stopping decisions, whereas its effect on click prediction was minor.With GPT-3.5, performance decreased but remained better than the context-removed variant, indicating effective use of context.Source: §4.2.2 · p. 4, Figure 2 · p. 4

5 CONCLUSION AND FUTURE WORK

The paper concludes that USimAgent is effective for simulating search queries and offers competitive click and stopping prediction on real user behavior data. It identifies combining LLMs with broader datasets as a future direction to improve predictive accuracy.

5 CONCLUSION AND FUTURE WORK.

  • USimAgent outperforms existing methods in query generation and is comparable to traditional methods for predicting clicks and stopping behaviors.These findings come from empirical investigation on a real user behavior dataset.Source: §5 · p. 4
  • Because zero-shot predictive accuracy may lag behind models trained on larger datasets, combining LLMs with broader data is proposed as future work.The limitation concerns predictive accuracy in zero-shot scenarios.Source: §5 · p. 4

Related papers

Loading 2403.09142v2…