PaperlayerView exact revision on arXiv

Source-linked AI summary

It Takes Two to Match: Co-Evolving Generative Retriever with Reinforcement Learning

Runpeng Dai, Kaili Huang, Changsung Kang, Ciya Liao

arXiv:2609.00638v1cs.IRcs.CL

TL;DR

CoGR targets the gap that LLM-based retrieval typically improves only the query side while leaving matching to a separate retriever. It generates keyword representations for both queries and items, trains them with aligned SFT and alternating GRPO against frozen opposite-side indexes, and improves F1 over the strongest baseline by 10.9% and 36.1% on two benchmarks. The analysis also reports stable co-evolution and increasingly aligned keyword spaces.

  • Problem

    LLM retrieval methods typically train only one side, usually queries, while relying on a separate downstream retriever for matching.

  • Method

    CoGR trains separate LLMs to generate query and item keywords, initializes them with SFT, and alternately optimizes them with GRPO against frozen opposite-side indexes.

  • Results

    CoGR achieves the best overall retrieval performance on the internal APP Marketplace and public WANDS benchmarks, improving F1 over the strongest baseline by 10.9% and 36.1%, respectively.

  • Takeaways & Limitations

    The results demonstrate the feasibility of jointly adapting query- and item-side keyword representations through retrieval feedback while retaining keyword-matching infrastructure.

  • Takeaways & Limitations

    The current ranking stage uses BM25 over generated keywords, and future work could extend rewards to downstream business objectives or design stronger ranking.

Abstract

from arXiv · show

Retrieval is the first stage of modern search and advertising systems, selecting a candidate set from a large item universe for downstream ranking and auction. Recent work increasingly leverages LLMs to improve retrieval through query expansion, data synthesis, and retrieval-feedback training. However, the generative component is typically used for query-side augmentation, while final matching is still delegated to a downstream retriever. We introduce CoGR, a retrieval framework that instead trains LLMs to directly construct retrieval representations on both query and item sides. Each generator produces a compact set of keywords, which are matched directly through an inverted index, preserving compatibility with existing keyword-based retrieval infrastructure. CoGR uses a two-stage training pipeline. Supervised fine-tuning first establishes an aligned keyword space, after which co-evolving reinforcement learning alternately optimizes the query- and item-side generators with GRPO against the opposite side's frozen index. Both sides optimize the same query-to-item retrieval $F_1$ objective: the query side receives retrieval $F_1$ directly, while the item side receives a counterfactual marginal reward measuring the change in query-side $F_1$ caused by its generated keywords. Across 10 representative sparse, dense, and generative baselines, CoGR achieves the best performance on both an internal APP Marketplace dataset and the public WANDS benchmark, improving $F_1$ over the strongest baseline by $10.9\%$ and $36.1\%$, respectively. Further analysis shows stable co-evolution and increasingly aligned query--item keyword spaces over training.

1 Introduction

CoGR addresses the limitation that LLM-based retrieval usually augments only queries while a separate retriever performs matching, by generating keyword representations for both queries and items. It combines aligned supervised initialization with co-evolving reinforcement learning and achieves the best reported performance on both evaluated datasets.

  • Retrieval selects candidates before downstream ranking and auction, so missed relevant items cannot be recovered and irrelevant candidates increase later-stage burden.
  • Existing LLM retrieval methods commonly improve query representations but still rely on a separate downstream retriever for matching.
  • CoGR trains separate query- and item-side LLMs to generate compact keyword sets that are matched directly through an inverted index.
  • CoGR first uses supervised fine-tuning to align the keyword space, then alternately optimizes both generators with GRPO against a frozen opposite-side index.
  • 10.9% and 36.1%: CoGR improves F1 over the strongest baseline on the internal APP Marketplace and public WANDS benchmarks, respectively.

2 Method

CoGR generates keyword-based retrieval representations for both queries and items, then trains the two generators through aligned supervised fine-tuning and alternating reinforcement learning. Its rewards optimize query-to-item retrieval F1 while keeping the opposite-side index fixed during each update.

  • 2.1 Overview: For each query and item, separate generators produce keyword sets whose overlap determines retrieval, with BM25 ranking the retrieved items.
  • 2.1 Overview: CoGR has two stages: supervised fine-tuning initializes both generators, followed by reinforcement learning that alternately updates the query- and item-side generators.
  • 2.2 Phase 1 — Supervised Fine-Tuning: SFT aligns the keyword spaces and provides enough initial recall for reinforcement learning to receive informative reward signals.
  • 2.3 Phase 2 — Co-Evolving Reinforcement Learning: Alternating optimization freezes the opposite-side index, allowing each generator to adapt to the latest keyword space of the other side.
  • 2.3 Phase 2 — Co-Evolving Reinforcement Learning: During query-side RL, sampled keyword sets retrieve items from a frozen item index and receive retrieval F1 as the GRPO reward.
  • 2.3 Phase 2 — Co-Evolving Reinforcement Learning: During item-side RL, replacing one item's reference keywords creates a counterfactual index whose change in aggregate query-side F1 measures that item's marginal contribution.
  • 2.3 Phase 2 — Co-Evolving Reinforcement Learning: The standard F1 reward can be replaced by a weighted F-measure when precision and recall have different business priorities.

3 Experiment Setting

The experiments evaluate CoGR on two industrial search datasets against sparse, dense, and generative retrieval baselines. The setup uses binary relevance labels, query-dimension splits with the full item universe retained, and separate query- and item-side generators.

  • 3.1 Datasets: Experiments use an internal APP marketplace dataset and the public WANDS product-search dataset.Internal items are applications, while WANDS items are products; both use title and description representations.
  • 3.1 Datasets: Categorical query–item relevance annotations are binarized into relevant and irrelevant classes for evaluation.Internal labels rated acceptable or better are relevant; WANDS Exact and Partial labels are relevant.
  • 3.1 Datasets: The experiment reports dataset statistics for the Internal and WANDS collections.These statistics are presented in Table 1.
  • 3.2 Baselines: CoGR is compared with sparse, dense, and generative retrieval families using representative established methods.Baselines include BM25 and SPLADE-v2, DPR and ANCE variants, and DSI, DSI-QG, RIPOR, and DeepRetrieval.
  • 3.3 Implementation: The method uses separate query- and item-side generators with the same backbone architecture at each model scale.CoGR directly generates keywords without chain-of-thought prompting and uses an SFT phase followed by GRPO-based co-evolving reinforcement learning.

4 Experimental Results

CoGR achieves strong retrieval performance through co-evolving query- and item-side optimization, with performance improving as training proceeds and keyword spaces becoming more specific and balanced. Ablations show that the full training design and richer generator inputs contribute to retrieval quality.

  • 4.1 Main Results: CoGR obtains the best overall F1 on both datasets, with scores of 0.396 and 0.682, respectively.Table 2 compares precision, recall, F1, MRR, and NDCG across retrieval methods and cutoffs.
  • 4.1 Main Results: Jointly adapting query and item keyword spaces outperforms optimizing only the query-side representation.The query-only variants keep item-side representations fixed, whereas the full model co-evolves both sides.
  • 4.2 Co-evolving Dynamics: Evaluation F1 increases from approximately 0.16 before co-evolving RL to approximately 0.40 after five alternating rounds.The largest gain occurs in the first round, followed by smaller but steady improvements.
  • 4.4 Analysis of Keyword Evolution: Co-evolving RL makes generated keywords more specific, reducing unigrams from 37% to 13% and increasing phrases of at least three words from 12% to 31%.Removed keywords are often generic, while newly introduced keywords tend to be longer and more informative.
  • 4.4 Analysis of Keyword Evolution: The query- and item-side vocabularies converge to a similar number of unique keywords after approximately three to four training rounds.The item-side vocabulary contracts early while the query-side vocabulary expands, progressively balancing the two spaces.
  • 4.5 Ablation on Additional Information: Adding search results raises F1 to 0.4379, whereas removing item descriptions reduces F1 from 0.3963 to 0.3759.Search results primarily help resolve ambiguous, misspelled, entity-centric, or non-English queries before keyword generation.

5 Related Works

CoGR extends retrieval research by co-evolving LLM-generated lexical representations for both queries and items, with keywords serving directly as the retrieval index.

  • Sparse and Dense Retrieval: Classical sparse retrieval uses lexical matching and inverted indexes, whereas dense retrieval matches shared continuous representations through vector similarity.Sparse and dense methods remain widely used paradigms in large-scale retrieval systems.
  • Generative Retrieval: Generative retrieval replaces nearest-neighbor search with autoregressive generation of document identifiers, using numeric or lexical identifier designs.These approaches include semantic IDs, document titles, substrings, and learned lexical identifiers.
  • LLMs and RL for Retrieval: Earlier LLM-based retrieval methods generate query expansions, pseudo-documents, or rewritten queries that downstream sparse or dense retrievers consume.Recent reinforcement-learning approaches optimize query generators using retrieval-metric or downstream utility rewards.
  • Generative Retrieval: CoGR jointly trains query- and item-side keyword generators, unlike approaches that primarily optimize query representations.The generated keywords directly serve as retrieval representations and co-adapt through retrieval feedback.
  • CoGR: CoGR preserves keyword-based retrieval infrastructure while alternately optimizing both generators toward a shared retrieval objective.This formulation distinguishes CoGR from methods that use generation mainly for query-side augmentation or delegate matching to a separate retriever.

6 Conclusion

CoGR trains LLMs to construct keyword-based retrieval representations on both query and item sides, using aligned initialization and alternating GRPO to co-adapt them. Experiments report consistent gains over sparse, dense, and generative baselines, while the authors identify broader objectives and stronger ranking as future directions.

  • Conclusion: CoGR combines aligned SFT initialization with alternating GRPO optimization against a frozen opposite-side index under a shared retrieval F1 objective.The query and item keyword spaces progressively co-adapt during training.
  • Future Work: The framework currently uses relevance metrics such as F1, while future work may extend rewards to business objectives including irrelevant ads percentage and revenue gain.The authors present this as a direction for extending the reward beyond relevance metrics.
  • Future Work: The current ranking stage uses BM25 over generated keywords, and stronger retrieval ranking could further improve overall retrieval quality.This is the stated ranking-related boundary and future-work direction.
  • Conclusion: Experiments on the Internal APP Marketplace dataset and WANDS demonstrate consistent gains over strong sparse, dense, and generative baselines.Both datasets convert categorical query–item relevance labels into binary relevance labels for gold-set construction.

B Training Details

The training-details section documents the prompts used to fine-tune the query-side and item-side generators.

  • Training Details: Figure 5 presents the prompts used to fine-tune the query-side Gq and item-side Gi generators.The figure provides the prompt specifications for both generator roles.

B.2 Hyperparameters

The hyperparameter section specifies separate decoding settings for initialization, evaluation, index construction, and RL rollouts, plus GRPO settings for both optimization directions.

  • Decoding Hyperparameters: Default Qwen sampling hyperparameters are used for item-keyword initialization, evaluation, and index construction, while RL uses unconstrained stochastic sampling for exploration.Decoding settings for these modes are summarized in Table 5.
  • GRPO Hyperparameters: GRPO training uses 8 NVIDIA B200 GPUs, with query-side and item-side optimization settings summarized in Table 6.Table 6 follows the original verl configuration field names.

B.3 Efficient Implementation of Item-Side Reward

The item-side reward is computed efficiently by caching reference retrieval statistics and updating only queries whose match status changes under a sampled item representation. This avoids rebuilding the counterfactual index or rerunning retrieval over all queries.

  • Reference retrieval states cache retrieved-item counts, true-positive counts, and relevant-item counts for each query.These cached counts support direct computation of reference F1 scores.
  • A sampled item keyword set is matched once against the frozen query-side inverted index, while reference retrieval also records queries retrieving that item.The two query sets identify which queries may be affected by replacing the item representation.
  • Only affected queries contribute to the counterfactual reward because unaffected queries have identical reference and candidate F1 scores that cancel.For each affected query, replacing one item can only add or remove that item from the retrieved set.
  • Each item-side rollout therefore needs one frozen-index lookup and constant-time F1 updates instead of full index rebuilding and retrieval over the entire query set.The optimization exploits that a rollout changes the representation of only one item.

C Retrieval Metrics at Additional Cutoffs

Held-out query-side retrieval metrics are reported at cutoffs 10 and 1000 using the same method grouping as Table 2. Table 8 separately examines keyword changes across alternating rounds and with additional input information.

  • Table 7 reports held-out query-side precision, recall, and F1 at retrieval cutoffs 10 and 1000.The table uses the same method grouping as Table 2.
  • The best value in each Table 7 column within a dataset is shown in bold.
  • Table 8 compares generated keywords across alternating rounds and after adding online search hints or application descriptions.The query side receives online search hints, while the item side receives application descriptions.

E Baseline Methods

The paper compares three retrieval-baseline families using documented base models and matched query- and item-side information where applicable. Examples in Table 8 illustrate generated keywords across query and item prompts, including prompts with additional information.

  • Three families of retrieval baselines are compared, with each method's base model summarized in Table 9.DeepRetrieval uses Qwen3-4B-Instruct-2507 as its base model.
  • Trainable baselines follow their standard training setups while using the same applicable query- and item-side information as CoGR.This includes item title and description fields and aligned prompting inputs.
  • Retrieval cutoff tuning selects the training-F1-maximizing cutoff and applies it unchanged to validation, with checkpoints selected by validation F1.Unless otherwise specified, trained baselines use eight GPUs with distributed data parallelism.
  • Table 8 shows query-side keywords with and without online search hints and item-side keywords with and without application descriptions.The examples include app-related terms such as genealogy, compression, mindfulness, games, food, and wellness.
  • The keyword examples span applications such as games, food delivery, puzzles, diving, wellness, and nature simulation.
Loading 2609.00638v1…