PaperlayerView exact revision on arXiv

Source-linked AI summary

PailitaoMMSearch2.0 Technical Report: Scaling Generative Retrieval to Unified End-to-End Multimodal Search

Xiaohan Ye, Zihan Gong, Jingqian Zhao, Lianyu Du, Yunmeng Shu, Xu Chen, Zesheng Zhou, Renchun You, Bocheng Chen, Jinsong Lan, Haihong Tang, Bo Zheng

arXiv:2610.36958v1cs.IRcs.AI

TL;DR

Industrial generative retrieval models are often tied to scenario-specific interactions and narrow query forms and retrieval objectives. PailitaoMMSearch scales one image–text-to-SID system across data, intents, and serving scenarios. Its controlled online A/B test reports statistically significant gains in item page views, transactions, and gross merchandise value.

  • Problem

    Industrial generative retrieval models are commonly specialized to a fixed online scenario, while open-ended image–text product queries can express different shopping relations without specifying an intent.

    Source: §1 · p. 1

  • Method

    PailitaoMMSearch combines lifecycle-specific data workflows with CPT, mixed-budget SFT, and unified RL to train one model to generate product SIDs from image–text queries.

    Source: §1 · p. 2, §4.2 · p. 4, §5.2 · p. 5

  • Results

    +17.90% item page views, +27.04% transactions, and +8.94% gross merchandise value in a statistically significant controlled online A/B test on Taobao Pailitao.

    Source: §1 · p. 2

  • Takeaways & Limitations

    PailitaoMMSearch2.0 provides an industrial-scale blueprint spanning multimodal data synthesis and quality control, multi-stage training, and production serving.

    Source: §8 · p. 8

    The evaluation covers only three shopping relations in one e-commerce domain, and the online A/B test cannot attribute gains to individual components.

    Source: §8 · p. 8

Abstract

from arXiv · show

Generative retrieval models have been increasingly adopted in search and recommendation systems. However, industrial generative retrieval models are commonly specialized to a fixed online scenario: they learn from scenario-specific interactions, support a narrow set of query forms and retrieval objectives. We study how to remove these couplings and build a unified end-to-end multimodal search system for open-ended image--text queries, heterogeneous shopping intents, and online scenarios with different resource constraints. We present \textbf{PailitaoMMSearch}, which directly generates product Semantic IDs from a raw query image and free-form instruction and scales along three dimensions. \textbf{Data scaling} uses a reusable toolset that can be composed into intent- and lifecycle-specific workflows for cold-start synthesis, quality control, real-traffic refinement, and continuous data production. \textbf{Intent scaling} progressively equips one model with e-commerce product knowledge, multiple retrieval relations, and adjustable reasoning budgets through continual pre-training (CPT), mixed-budget supervised fine-tuning (SFT), and unified reinforcement learning (RL). The RL stage combines intent-balanced optimization and Sample--Beam rollouts aligned with multi-candidate serving. \textbf{Scenario scaling} exposes quality--cost operating points across model capacity and reasoning budget. The system is deployed in multimodal search applications on Taobao Pailitao. A controlled online A/B test reports statistically significant improvements, including item page views $\textbf{+17.90\%}$, transactions $\textbf{+27.04\%}$, and gross merchandise value $\textbf{+8.94\%}$.

1 Introduction

Industrial generative retrieval remains tied to particular online scenarios, limiting its supervision, retrieval relations, and resource configurations. PailitaoMMSearch formulates open-ended image–text retrieval as one model generating product SIDs, with data, intent, and scenario scaling; its online test reports significant business gains.

1 Introduction

  • Scenario-specific systems bind supervision to existing entry points, retrieval objectives to individual models or routing, and model and decoding choices to one service-level objective.These couplings can increase system complexity and maintenance cost, allow routing errors to propagate, and make configurations difficult to reuse across services.Source: §1 · p. 1
  • Open-ended image–text queries can express different relations, from preserving or editing a pictured product to retrieving a complementary item, without naming the intent.Query complexity also ranges from direct visual and attribute evidence to multi-step comparison of category, style, function, and constraints.Source: §1 · p. 1
  • The system directly maps a raw query image and free-form instruction to product SIDs, inferring the retrieval relation without explicit intent input or intent-specific serving routes.The paper frames this as a native-VLM system and an industrial-scale blueprint spanning data synthesis and quality control, training, and production serving.Source: §1 · pp. 1–2
  • Data workflows vary by intent and lifecycle; multi-stage training aligns retrieval relations in one model, while model capacity and reasoning budget define deployable quality–cost operating points.The training recipe combines continual pre-training, mixed-budget supervised fine-tuning, and unified reinforcement learning with Sample–Beam rollouts aligned to multi-candidate serving.Source: §1 · pp. 1–2
  • +17.90% item page views, +27.04% transactions, and +8.94% gross merchandise value in a controlled Taobao Pailitao online A/B test.The reported gains are statistically significant.Source: §1 · p. 2

2 Related Work

Related work develops generative retrieval and Semantic IDs, multimodal product retrieval, and model adaptation with adaptive reasoning. PailitaoMMSearch differs by scaling SID generation across query distributions, retrieval relations, and serving configurations.

2 Related Work

  • Generative retrieval decodes item identifiers from queries, with hierarchical residual-quantized tokens and later work adding collaborative signals and richer identifier construction [5] [18] [20] [9] [6] [29].Unified recommendation and industrial systems extend the formulation across tasks and production traffic [3] [7] [30].Source: §2 · p. 2
  • CLIP-style models support large-scale visual and cross-modal product retrieval, while native VLMs process pixels and language jointly; these methods often use vector similarity or predefined composed-retrieval tasks [16, 25, 28] [4] [21].PailitaoMMSearch instead generates catalog SIDs with one policy over overlapping relations expressed through unrestricted image–text instructions.Source: §2 · p. 2
  • CPT, instruction tuning, and RL adapt models to downstream objectives, while chain-of-thought and inference-time methods trade computation for reasoning quality [14] [17] [19] [13] [22] [23] [26].PailitaoMMSearch uses explicit prompt-controlled reasoning budgets and separates sampled reasoning from beam SID generation.Source: §2 · p. 2

3 Problem and System Overview

PailitaoMMSearch formulates retrieval as generating catalog identifiers from image–text input, then uses a shared policy and scenario-specific configuration under resource constraints. Its architecture separates how behaviors are learned from how they are deployed.

3.1 Unified Multimodal Generative Retrieval

  • A raw image, free-form instruction, and optional application context form the input; the model generates a candidate set of product SIDs, and decoding is the retrieval operation.Each catalog item has a three-level hierarchical SID, with 8,192 codebook entries at each level.Source: §3.1 · p. 2
  • The request’s retrieval relation is implicit and must be inferred from its image–text input; similar, editing, and matching are evaluation slices, not required serving inputs.The slices need not be mutually exclusive.Source: §3.1 · p. 2
  • The system seeks a shared policy and serving configuration that maximize retrieval utility subject to application-specific P95-latency and memory limits.The configuration comprises model scale, reasoning budget, and beam width; the formulation is system-level, not a single differentiable training loss.Source: §3.1 · p. 2

3.2 Overall Architecture

  • All intents share a native VLM, SID vocabulary, and decoding interface, while scenario scaling selects a configuration without changing the query interface or adding an intent-specific expert.Raw image patches and text tokens are jointly processed before an optional think block and product SIDs are emitted.Source: §3.2 · p. 3
  • Data scaling supplies quality-controlled supervision, intent scaling learns a shared policy, and scenario scaling evaluates and deploys it at different capacity and reasoning budgets.This order separates how a behavior is learned from how it is used under production constraints.Source: §3.2 · p. 3, Figure 1 · p. 3

4 Data Scaling with Composable Toolsets

Data scaling reuses configurable operations to build intent- and lifecycle-specific workflows, from cold-start synthesis through real-traffic refinement and continuous production. Quality control gates accepted data across differing source paths.

4.1 Data Sources and Lifecycle Challenges

  • Before launch, proxy relations provide training examples, while later traffic adds query–item interactions that are often sparse and noisy.Data scaling therefore targets volume, query and intent coverage, lifecycle coverage, and quality rather than one fixed pipeline.Source: §4.1 · p. 3

4.2 Reusable Toolset

  • Five reusable tools filter pairs, verify relations, generate or repair queries, add optional reasoning supervision, and apply terminal quality control.Cheap checks precede costly multimodal verification; tool semantics and rubric vary by intent, so adding an intent requires a relation specification and workflow configuration rather than a separate annotation system.Source: §4.2 · p. 4

4.3 Intent- and Lifecycle-Specific Workflows

  • Cold-start synthesis, real-traffic refinement, and continuous production compose the tools differently, while mandatory Quality-Control keeps accepted corpora comparable.The workflows respectively use proxy relations before launch, repair noisy observed interactions, and mix proxy sources with accumulated traffic for under-covered regions.Source: §4.3 · p. 4
  • In the editing example, the workflow preserves category and color while changing sleeve length, rejecting targets with unsupported extra differences.Matching uses a different rubric, where compatibility and context replace attribute preservation as key acceptance conditions.Source: §4.3 · p. 4

5 Intent Scaling via Multi-Stage Training

Intent scaling progressively adapts one model to e-commerce products, multiple retrieval relations, and different reasoning budgets. CPT, mixed-budget SFT, and unified RL build and optimize these capabilities, with sampled reasoning and beam-decoded product SIDs aligning rollouts to serving.

5.1 Continual Pre-Training

  • CPT adapts the VLM to e-commerce products and SID vocabulary through product, behavior, attribute, and general-language tasks.The stage provides a stronger initialization for the SID vocabulary space used in generative retrieval.Source: §5.1 · pp. 4–5

5.2 Multi-Intent, Mixed-Budget SFT

  • Mixed-budget SFT jointly trains one model on similarity, editing, and matching samples, with prompt-controlled modes from direct SID output to extended reasoning.At serving, the model receives no explicit intent label and infers the retrieval relation from the image and instruction; the reasoning mode remains specified through the system prompt.Source: §5.2 · p. 5

5.3 Intent-Balanced Unified RL

  • Unified RL applies intent-specific reward rubrics while β_t prevents high-volume intents from dominating optimization.GRPO compares outcomes for the same query, and all rollouts update the same policy rather than selecting intent-specific serving routes.Source: §5.3 · p. 5

5.4 Sample–Beam Rollouts

  • Sample–Beam rollouts sample reasoning prefixes, then beam-decode SID candidates to reduce the mismatch with production retrieval.Candidate-level rewards are normalized across beam branches before a GRPO-style clipped surrogate updates reasoning and SID tokens; because beam selection is deterministic, this is not an unbiased on-policy estimator.Source: §5.4 · p. 5

6 Scenario Scaling and Deployment

Scenario scaling offers operating points through model capacity and reasoning budget, then serves raw image–instruction queries as candidate SID sets. The system runs across Taobao Pailitao applications with different traffic and quality requirements.

6 Scenario Scaling and Deployment

  • Native VLMs at 0.8B, 2B, and 4B parameters provide coarse resource adaptation, while No-think, Low-think, and High-think offer finer operating points.Explicit reasoning budgets support predictable cost across latency, memory, and throughput constraints.Source: §6 · p. 5
  • At serving, the selected model takes a raw image and instruction, optionally reasons, then beam-decodes product SIDs into catalog items without an intent classifier.Applications choose model size, reasoning policy, and beam configuration for their resource and latency envelope; these settings are held fixed in online comparisons.Source: §6 · p. 5
  • PailitaoMMSearch2.0 serves Multimodal Search, Find Matching Tab, and Find Similarity Tab, with sampled online interactions refined before subsequent training.The applications share request, model, and SID interfaces while operating under different traffic and quality requirements.Source: §6 · p. 5

7 Experiments

The experiments assess retrieval quality across three intents, data and training choices, and model-and-reasoning operating points. Controlled online tests report significant business gains after deployment.

7.1 Setup

  • Evaluation used later-window automatic tests, duplicate-free splits, disjoint human evaluator calibration, and macro-averaged GR, ER, and NDCG@10 across three intents.Controlled experiments generally used the 2B model with High-think mode and held evaluation and training settings fixed within each ablation.Source: Table 2 · p. 6, Table 3 · p. 6, §7.1 · p. 6

7.2 Overall Multi-Intent Retrieval Quality

  • 2B Low-think led Pailitao variants with macro NDCG 0.918 and macro ER 55.1%, at 278 ms on one GPU.It improved macro ER over Qwen3.8-Flash-Next, GPT6-luna, and the internal pipeline by 2.9, 3.3, and 5.6 points, respectively.Source: §7.2 · p. 6, Table 4 · p. 7

7.3 Effects of Data Scaling

  • Accepted-data scaling raised macro ER from 46.32% to 49.03% (+2.71 points); removing real-traffic supervision and quality control lowered it further.Similar retrieval gained 5.08 points from data scaling; removing real-traffic data lowered macro ER by 0.78 points, and further removing quality control lowered it by another 0.28 points.Source: §7.3 · p. 6, Table 5 · p. 7

7.4 Effects of Intent Scaling

  • CPT improved macro GR by 0.8 points and macro ER by 1.2 points under the same downstream SFT configuration.This comparison used a Base VLM and a model continually pre-trained for e-commerce product understanding.Source: Table 6 · p. 7, §7.4 · p. 7
  • Adding similar supervision raised editing ER from 33.11% to 35.88% (+2.77 points); adding matching yielded matching ER 66.09% and slightly lowered editing ER.The three-intent mixture raised similar ER from 44.63% to 45.56% (+0.93 points), while editing ER remained 2.32 points above the editing-only setting.Source: Table 7 · p. 7, §7.4 · p. 7
  • Sample–Beam RL exceeded SFT by 5.97 macro-ER points, with gains of 4.09, 5.05, and 8.79 points on similar, editing, and matching retrieval.Standard unified RL raised macro ER from 49.03% to 52.70% (+3.67 points), and Sample–Beam added a further 2.30 points.Source: Table 8 · p. 7, §7.4 · p. 7

7.5 Effects of Scenario Scaling

  • The best scenario operating point depends on quality–latency needs: 4B High-think reached macro ER 55.41% at 1,133 ms RT.For 2B, Low-think achieved macro ER 55.13% at 278 ms; High-think raised matching ER to 74.88% but nearly doubled response time to 544 ms.Source: §7.5 · pp. 7–8, Table 9 · p. 8, Figure 2 · p. 8

7.6 Online A/B Tests in Production

  • Item page views +17.90%, transactions +27.04%, and GMV +8.94% in controlled A/B tests were statistically significant (p<0.05).Online measurements evaluate the complete deployed configuration; the system was rolled out to 100% of traffic in the corresponding production scenarios.Source: §7.6 · p. 8

8 Conclusion, Limitations, and Future Work

PailitaoMMSearch2.0 unifies free-form image–text search across shopping relations, with a composable data pipeline and multi-stage training. The study reports production A/B gains while identifying limits in relation coverage, model-size evaluation, offline judging, and component attribution.

Conclusion

  • The system combines composable data production with CPT, mixed-intent SFT, and unified RL, and online A/B tests demonstrate system-level gains in production.Offline experiments also show benefits from data scaling, domain-adaptive CPT, cross-intent supervision, and Sample–Beam RL.Source: §8 · p. 8

Limitations

  • Evaluation covers only three shopping relations in one e-commerce domain, and the separate model-size checkpoints leave their interactions underexplored.The online A/B test measures the deployed system as a whole and cannot attribute gains to individual components.Source: §8 · p. 8

Future Work

  • Future work will broaden relation coverage, jointly optimize retrieval quality against latency and token cost, and study adaptive or distilled models sharing parameters across serving budgets.The authors also plan evaluator ensembles with periodic human audits and stability tests under catalog and traffic drift.Source: §8 · p. 8

A Evolution of the Editing-Intent Rubric

The appendix refines an automatic relevance rubric for editing-oriented retrieval, requiring category fit and adherence to explicit edit and preservation requirements. It describes how the rubric is scored and applied in data production.

A Evolution of the Editing-Intent Rubric

  • Editing queries specify a controlled transformation of the query product, such as changing sleeve length while preserving color.The evaluator checks fine-grained category fit, explicit editing and preservation requirements, and consistency with the requested direction.Source: A Evolution of the Editing-Intent Rubric · p. 10

A.1 Rubric Refinement

  • Held-out human-annotation error analysis found visual and semantic similarity insufficient when targets missed the requested edit or violated a preservation requirement.These errors motivated revisions summarized in Table 10.Source: §A.1 · p. 10

A.2 Final Scoring Rule

  • The final rubric identifies the fine-grained target category, extracts explicit edit and preservation requirements, and evaluates evidence from both image and title.It assigns one of four scores.Source: §A.2 · p. 10
  • A clearly red long-sleeve shirt matching the requested category scores 3, while obscured sleeve evidence may score 2 if no requirement is contradicted.A blue long-sleeve shirt scores 1 for violating color preservation; a jacket or unrelated product scores 0 for category mismatch.Source: §A.2 · p. 10
  • Targets scoring 2 or 3 pass the production relevance gate, while score 3 contributes to the reported Excellent Rate.The tables show the decision structure, not the complete production prompt.Source: §A.2 · p. 10
  • The complete production prompt also includes intent-specific demonstrations and output-format instructions beyond the tables’ decision structure.The tables therefore do not reproduce the full prompt used in production.Source: §A.2 · p. 10

Related papers

Loading 2610.36958v1…