PaperlayerView exact revision on arXiv

Source-linked AI summary

PailitaoGR: Latent Think-with-Images for Generative Image Retrieval

Xiaomeng Fan, Yueran Liu, Shengyu Zhou, Chenghan Fu, Wanxian Guan, Feng Li, Chuan Yu, Jian Xu, Bo Zheng

arXiv:2608.26658v1cs.CVcs.AIcs.IR

TL;DR

Real-world image-search queries mix target objects, useful auxiliary cues, and irrelevant content, making target-focused and selective perception necessary for generative retrieval. PailitaoGR internalizes these capabilities in a model using only the original query image, and it outperforms the directly SFT-trained baseline by an average of 13.8% across ten metrics. The results support Zooming without Cropping and Reading without OCR through selective capability internalization.

  • Problem

    Real-world query images contain search targets, useful auxiliary evidence, and irrelevant visual content, requiring generative retrieval to focus on targets and selectively use relevant evidence.

  • Method

    PailitaoGR internalizes target-focused perception and selective auxiliary-evidence utilization through target and auxiliary enhancers, distillation, and attention guidance while using only the original query image.

  • Results

    13.8% average improvement over the directly SFT-trained baseline across ten metrics; PailitaoGR also surpasses the Crop Teacher by 4.76% and the OCR Teacher by 2.76%.

  • Takeaways & Limitations

    PailitaoGR achieves Zooming without Cropping and Reading without OCR, indicating that target focusing and useful auxiliary-evidence utilization can be internalized for full-image retrieval.

  • Takeaways & Limitations

    The industrial dataset contains anonymized user behavior logs, search queries, and item images handled under platform-defined retention, access-control, and deletion policies.

Abstract

from arXiv · show

Generative retrieval has demonstrated strong performance by directly generating product semantic identifiers (SIDs). Extending this paradigm to image search, however, is nontrivial because real-world query images contain diverse information, including the search target, useful auxiliary evidence, and irrelevant visual content. This requires the model to identify and focus on the search target while selectively utilizing auxiliary evidence. In this paper, we propose \textbf{PailitaoGR}, a \emph{Latent Think-with-Images} method for generative image retrieval, which internalizes target-focused perception and selective auxiliary-evidence utilization into a the generative retrieval model, enabling \textit{Zooming without Cropping} and \textit{Reading without OCR}. Specifically, we design a target-focused perception mechanism that identifies and enhances visual tokens of the search target, consisting of a target Enhancer and a learning strategy based on on-policy distillation and attention guidance loss, enabling the model to focus on search-target regions. We also design a selective auxiliary-evidence utilization mechanism that identifies and enhances visual tokens of auxiliary evidence, including an auxiliary enhancer and an in-capacity incremental contrastive distillation strategy, enabling the model to exploit auxiliary evidence. We construct training and validation sets sampled from real-world online image-search logs. Experiments show that our method outperforms existing baselines by an average of 13.8\%, validating its effectiveness.

1 Introduction

PailitaoGR addresses the difficulty of retrieving target items from cluttered query images by focusing on the search target and selectively using auxiliary evidence. It internalizes these capabilities in a generative retriever, achieving strong performance without relying on extra tool calls.

  • Motivation: Real-world query images combine search targets, useful auxiliary cues, and irrelevant content, requiring both target focusing and selective evidence use.Examples of auxiliary cues include brand and model information, while irrelevant content includes watermarks and other objects.
  • Motivation: Tool-assisted visual retrieval can add computation and multi-step inference, while irrelevant or inaccessible tool-derived information can harm capability transfer.These issues conflict with online retrieval latency requirements and limited perceptual capacity.
  • Approach: PailitaoGR internalizes target focusing and auxiliary-evidence utilization while operating solely on the original query image.Its two mechanisms are described as Latent Think-with-Images capabilities for generative image retrieval.
  • Approach: Target-focused perception uses a target enhancer, Crop Teacher, on-policy distillation, and attention guidance to center SID generation on search-target regions.The design is intended to provide Zooming without Cropping.
  • Approach: Selective auxiliary-evidence utilization uses an auxiliary enhancer and incremental contrastive distillation to exploit useful textual evidence while avoiding irrelevant or conflicting information.The OCR Teacher supplies cropped target regions and pre-extracted OCR text, while supervision is activated selectively when textual information benefits SID prediction.
  • Evaluation: Training and validation sets come from Pailitao online image-search logs, and PailitaoGR achieves the best performance against existing generative and contrastive retrieval methods.The paper also reports consistent gains over Crop Teacher and OCR Teacher despite using only the original query image.

2 Related Works

Related work spans representation-based vision search, generative retrieval with semantic identifiers, and methods for text, image, and multimodal search. Prior industrial vision-search systems provide important context for PailitaoGR’s full-image retrieval setting.

  • Representation-based vision search: Representation-based vision search encodes query images and products into dense representations and retrieves products through nearest-neighbor search.Traditional e-commerce systems use dual-flow architectures for cross-modal alignment between visual and textual product content.
  • Generative retrieval: Generative retrieval reformulates retrieval as autoregressive generation of product semantic identifiers conditioned on the input query.SIDs are commonly constructed with clustering- or quantization-based methods such as RQ-VAE and FSQ.
  • Text-based generative search: Text-based generative search improves semantic alignment, query understanding, or latent user-intent modeling through several complementary directions.Examples include identifier construction, contrastive constraints, context-aware reasoning, self-distilled thought augmentation, and category-guided latent intent learning.
  • Image and multimodal search: Generative retrieval has also been extended to image and multimodal search through frameworks including IRGen, GENIUS, and OneVision.OneVision is described as an industrial e-commerce vision-search framework whose generative retriever takes cropped query images as input.

3 Method

PailitaoGR formulates image retrieval as SID generation and internalizes target-focused perception plus selective auxiliary-evidence utilization while operating on the original query image.

  • 3.1 Formulation: Generative image retrieval generates high-probability candidate SIDs autoregressively and maps the top-K SIDs to products.Each product is assigned a sequence of discrete semantic tokens called an SID.
  • 3.1 Formulation: PailitaoGR addresses heterogeneous query images by focusing on the search target and selectively exploiting complementary evidence for fine-grained item identification.The model operates solely on the original query image through Target-Focused Perception and Selective Auxiliary-Evidence Utilization.
  • 3.2.1 Target Enhancer: The Target Enhancer scores visual-token relevance to the search target and applies token-level residual modulation to enhance selected tokens.The residual connection preserves the original visual information while the learnable residual controls enhancement magnitude.
  • 3.2.2 Target-Focused Perception Objective: On-policy distillation queries the Crop Teacher and student under the same student-generated prefixes, reducing the mismatch between training and autoregressive inference.The teacher’s top-K candidate distributions are compared with the student’s distributions using Jensen–Shannon divergence.
  • 3.2.2 Target-Focused Perception Objective: Granularity-aware attention guidance directs SID-token attention toward target regions and regulates its concentration from broad coarse-token attention to focused fine-token attention.ROT constraints identify where visual evidence should be gathered, while entropy guidance controls how broadly it is collected at each semantic granularity.
  • 3.3 Selective Auxiliary-Evidence Utilization Mechanism: Selective Auxiliary-Evidence Utilization uses an auxiliary enhancer and selective distillation to identify useful auxiliary information while avoiding irrelevant or misleading evidence.The mechanism determines which auxiliary capability should be internalized, and the overall objective supports target-centered SID generation from the original query image.

4 Data Construction

The dataset is built from large-scale online image-search logs, filtered by behavioral evidence and organized across seven product categories. Training and test queries are separated temporally, with additional behavioral and verification requirements for testing.

  • The dataset uses online image-search logs from the Pailitao e-commerce platform, retaining queries associated with behavioral SIDs.Retained behaviors include clicks, purchases, add-to-cart actions, and favorites.
  • Training queries must have more than three behavioral SIDs and come from seven frequent product subcategories.The categories are women’s clothing, children’s clothing, men’s clothing, women’s shoes, men’s shoes, digital products, and furniture.
  • The test set is temporally disjoint from training, requires more than three behavioral SIDs and at least one purchase, and is verified by humans and matching models.
  • The training and test sets contain 1,159,746 and 8,647 query images, respectively, covering seven product categories.Each query is associated with multiple positive items, with an average of 5.87 positive items per query in the training set passage.

5 Experiments

Experiments evaluate retrieval performance, target focusing, OCR-cue utilization, attention behavior, and the contributions of PailitaoGR’s core components. PailitaoGR consistently improves over baselines and teacher models, with stronger gains for smaller targets and queries containing OCR.

  • Main Results: 12.16% average improvement over the second-best method across ten metrics gives PailitaoGR the best overall retrieval performance.Under the same crop-based setting, PailitaoGR consistently outperforms IRGen and GENIUS.
  • Main Results: 13.8% average improvement over the directly SFT-trained baseline across ten metrics demonstrates the effectiveness of capability internalization.PailitaoGR also surpasses the Crop Teacher by 4.76% and the OCR Teacher by 2.76% across the ten metrics, despite using only the original query image.
  • Target-Focusing Capability Analysis: PailitaoGR improves consistently across search-target ROT groups, with more pronounced gains when the target occupies a smaller query-image region.This validates its ability to focus on the search target in complex scenes without cropping.
  • OCR Reading Capability Analysis: PailitaoGR achieves more evident improvements on queries containing OCR while remaining competitive on queries without OCR.The pattern supports exploiting useful textual cues from the original image without indiscriminate reliance on them.
  • Target-Focusing Capability Analysis: The SFT baseline shows dispersed attention affected by background regions, whereas PailitaoGR concentrates more attention on the search target.Attention is averaged across heads and Transformer layers, aggregated across three SID levels, and projected back to the original image.
  • Ablation Studies: Both target-focused perception and selective auxiliary-evidence utilization consistently improve retrieval performance when progressively added to the SFT baseline.The ablation isolates gains from search-target focusing and auxiliary-evidence utilization.
  • Ablation Studies: The full target-focused perception model performs best, with on-policy distillation providing the largest gain and ROT attention guidance and entropy regularization adding complementary improvements.These objectives respectively transfer target-focused prediction behavior, provide explicit target-region supervision, and encourage coarse-to-fine attention concentration.
  • Ablation Studies: Filtering auxiliary evidence by utility improves four metrics by about 2.0 percentage points on average, while accessibility filtering brings further consistent improvements.The results support transferring auxiliary capabilities only when evidence is both useful and accessible to the student.

6 Conclusion

PailitaoGR brings Latent Think-with-Images into full-image retrieval by internalizing target-focused perception and selective auxiliary-evidence utilization. It strengthens target and relevant auxiliary cues, enabling Zooming without Cropping and Reading without OCR while outperforming existing methods and teacher models using the original query image.

  • PailitaoGR internalizes target-focused perception and selective auxiliary-evidence utilization for full-image generative retrieval.
  • The target-focused mechanism emphasizes target-related visual tokens and guides SID generation toward discriminative target regions, enabling Zooming without Cropping.
  • The auxiliary-evidence mechanism selectively enhances cues relevant to the search target and beneficial to SID prediction, enabling Reading without OCR.
  • Training and evaluation data are constructed from real-world online image-search logs.
  • PailitaoGR consistently outperforms existing retrieval methods and surpasses both Crop Teacher and OCR Teacher using only the original query image.

7 Ethical Considerations

The industrial dataset uses anonymized user behavior logs, search queries, and item images under purpose-limited research and production use. Platform governance, access controls, retention, and deletion policies provide safeguards for model training and evaluation.

  • The industrial dataset contains anonymized user behavior logs, search queries, and item images.
  • The data are used only for stated research and production purposes without attempting to identify individual users.
  • Data handling follows platform-defined policies for retention, access control, and deletion.
  • Anonymization, purpose limitation, and platform-level governance provide safeguards for responsible model training and evaluation.
Loading 2608.26658v1…