PaperlayerView exact revision on arXiv

Source-linked AI summary

Progressive Learning from Content to Relevance to Personalization for CTR Prediction

Zhiguang Yang, Yulin Yang, Fuyu Lv, Dan Ou

arXiv:2610.29138v1cs.IRcs.MM

TL;DR

ID-based CTR models have difficulty using fine-grained multimodal content, despite its relevance to item, query, and user preferences. FineSID tokenizes content into fine-grained discrete IDs, pretrains a shared predictor on query–item relevance, and fine-tunes it on clicks; evaluations report gains in CTR and within-product creative discrimination.

  • Problem

    Integrating fine-grained heterogeneous content into ID-based CTR predictors remains challenging, while coarse discrete quantization can discard spatial structure and textual detail.

    Source: §1 · p. 1

  • Method

    FineSID progressively tokenizes multimodal content into fine-grained SIDs, pretrains a shared predictor on query–item relevance, and fine-tunes it with user click supervision.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    FineSID improves standard product-level CTR prediction and within-product creative discrimination, with reported offline gains and positive online changes in clicks and GMV.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    FineSID’s results indicate that fine-grained multimodal content can support CTR prediction and creative discrimination beyond conventional ID memorization.

    Source: §1 · p. 2, §7 · p. 9

    The reported negligible serving overhead is for late-stage ranking; extending fine-grained tokenization to extreme low-latency early-stage retrieval remains a boundary for the approach.

    Source: §6 · p. 9

Abstract

from arXiv · show

Multimodal click-through rate (CTR) prediction is central to search systems, where ranking depends on item content, query relevance, and user preference. Existing approaches incorporate content through dense embeddings or coarse discrete IDs, yet effectively integrating fine-grained heterogeneous content with ID-based predictors remains challenging. We propose FineSID, a progressive learning framework that unifies content tokenization, query--item semantic pretraining, and personalized post-training via a shared fine-grained discrete representation. Each stage optimizes an explicit objective while preserving token-level access to content. First, a pretrained multi-scale visual tokenizer and text tokenizers convert heterogeneous content into fine-grained Semantic IDs (SIDs), explicitly aligning them with the ID features. Second, full-parameter semantic pretraining under query--item relevance supervision establishes a robust matching prior, jointly optimizing the interactions of multimodal SIDs and conventional fields within a shared predictor. Third, this initialized predictor is directly fine-tuned with user click supervision to capture implicit personalized preferences. Extensive offline and online evaluations demonstrate substantial gains in both standard CTR prediction and core business metrics. Furthermore, FineSID improves within-product creative discrimination, demonstrating its semantic generalization beyond conventional ID memorization.

1 INTRODUCTION

FineSID addresses the challenge of integrating fine-grained multimodal content into ID-based CTR prediction through progressive learning from item content to query relevance and user personalization. Evaluations report gains in CTR prediction, cold-start scenarios, within-product creative discrimination, and online business metrics.

  • Sparse-ID CTR models struggle with cold-start items and do not directly use multimodal semantic knowledge from vision-language models.Integrating multimodal content into ID-based models remains challenging, while coarse quantization can discard spatial structure and textual detail.Source: §1 · p. 1
  • FineSID progresses from fine-grained item content tokenization to query–item relevance learning and user-click personalization in a shared ID-based predictor.It represents multimodal content with fine-grained Semantic IDs (SIDs).Source: §1 · pp. 1–2
  • FineSID’s three stages tokenize content, train on query–item relevance, then fine-tune with user click supervision.Multi-scale visual and source-specific text tokenizers preserve token-level access to content.Source: §1 · pp. 1–2
  • FineSID reports consistent Taobao AUC and GAUC improvements, cold-start gains, within-product creative discrimination, and positive online changes in clicks and GMV.The creative comparison holds the product ID fixed, indicating discrimination of content differences beyond conventional ID memorization.Source: §1 · p. 2

2 PROBLEM FORMULATION

The formulation treats an exposure as a user, query, item, and content combination, with the ultimate task of predicting clicks. FineSID first trains shared parameters on query–item relevance, then retains and fine-tunes the shared network for personalized CTR.

  • An exposure x contains user and context features, a search query, an item ID, and fine-grained multimodal content.Different creatives can belong to the same item.Source: §2 · p. 2
  • The ultimate prediction task is whether the user clicks, represented as a binary CTR outcome.Source: §2 · p. 2
  • FineSID first pretrains shared parameters using query–item relevance supervision, then retains the input interface and shared network for click-supervised CTR fine-tuning.Source: §2 · p. 2
  • Evaluation distinguishes product-level ranking across different item IDs from creative discrimination when the item ID is fixed and displayed content varies.Source: §2 · p. 2

3 FINESID

FineSID integrates multimodal content into ID-based CTR prediction through a shared feature-interaction network and three progressive learning stages. Fine-grained discrete tokens feed semantic relevance pretraining, followed by click-supervised personalized fine-tuning.

3 FINESID

  • FineSID uses a shared feature-interaction architecture in semantic pretraining and personalized post-training, with tokenization, relevance learning, and CTR fine-tuning as its three stages.Source: §3 · p. 3

3.1 STAGE I: FINE-GRAINED CONTENT TOKENIZATION

  • Stage I represents images with 30 spatially indexed, coarse-to-fine visual SIDs and retains field-reserved text token sequences for token-level multimodal access.Visual codes are extracted offline; text sequences preserve source identity and term position.Source: §3.1 · pp. 3–4

3.2 SHARED FEATURE INTERACTION ARCHITECTURE (θ)

  • Ordered fusion combines base-field embeddings, multi-scale visual representations, and query-conditioned text embeddings before latent-slot attention captures cross-feature interactions.The backbone projects the concatenated features into a 768-dimensional global state to avoid standard self-attention over thousands of raw tokens and ID fields.Source: §3.2 · p. 4

3.3 STAGE II: FULL-PARAMETER SEMANTIC PRETRAINING

  • Stage II trains the shared predictor on query–item relevance to establish semantic alignment, with teacher-provided soft relevance distributions as targets.Relevance gradients update the shared trainable parameters, including SID embeddings, categorical field lookups, interaction blocks, and normalization parameters.Source: §3.3 · pp. 4–5

3.4 STAGE III: PERSONALIZED CTR POST-TRAINING

  • Stage III initializes CTR training from the relevance-pretrained shared parameters and normalization states, then fine-tunes the network with observed user clicks.The relevance head is replaced by a CTR output layer, and all shared predictor parameters are transferred and fine-tuned.Source: §3.4 · p. 5

4 EXPERIMENTS

The experiments evaluate FineSID on standard product-level CTR prediction and within-product creative discrimination, comparing it with ID-based and content- or relevance-aware baselines. Taobao offline results and ablations assess overall gains and the contributions of the framework’s components.

4 EXPERIMENTS

  • The experiments test whether FineSID improves both product-level CTR prediction and within-product creative discrimination.The Taobao dataset uses a chronological split: August 25–31, 2026 for CTR training and September 1, 2026 for offline evaluation.Source: §4 · p. 5, §4.1 · p. 5

4.2 EVALUATION METRICS

  • Product-level evaluation reports AUC and impression-weighted GAUC; cold-start items have fewer than 1,000 evaluation-day page views, and creative ranking uses NSCTR.For Taobao, AUC and GAUC are reported as percentage-point improvements and NSCTR as a relative percentage; MMCTR results report absolute AUC.Source: §4.2 · p. 6

4.3 COMPARISON METHODS

  • FineSID is compared with an ID-based baseline and methods using continuous content embeddings, visual or residual-quantized IDs, semantic IDs, relevance supervision, or joint relevance and CTR training.The listed comparison methods include HiFormer, LEARN-style embedding, Visual ID, RQ-VAE SID, COINS, PRECTR, and Joint relevance + CTR.Source: §4.3 · p. 6

4.4 TAOBAO OFFLINE EVALUATION

  • FineSID achieves the largest reported gains over HiFormer plus ID features: +0.25 AUC and +0.26 GAUC points overall, +0.38 and +0.34 points on cold-start items, and +0.75% NSCTR.Table 2 reports AUC and GAUC changes in percentage points and NSCTR as a relative percentage improvement; cold-start items have evaluation-day PV < 1000.Source: Table 2 · p. 6, §4.4 · p. 6

4.5 ABLATION STUDY

  • Removing Stage II semantic pretraining lowers overall and cold-start metrics, with NSCTR improvement falling from 0.75% to 0.51%.The ablation passage also reports that removing fine-grained SIDs yields a 0.03-point decrease in cold-start AUC versus HiFormer and only +0.02% NSCTR.Source: §4.5 · pp. 6–7, Table 3 · p. 7
  • Replacing query-conditioned cross-attention and multi-scale slots with flat average pooling reduces AUC gains over HiFormer from 0.25 to 0.12 points.The reported comparison supports spatially indexed visual features and query-conditioned text embeddings over flat average pooling.Source: §4.5 · p. 7, Table 3 · p. 7

4.6 PUBLIC BENCHMARK: MMCTR

FineSID improves performance on the public MMCTR benchmark even without relevance pretraining, and its within-product and online evaluations report gains in creative discrimination and business metrics.

4.6 PUBLIC BENCHMARK: MMCTR

  • +0.77 AUC points over HiFormer was the largest gain among reported methods on MMCTR, using FineSID without relevance pretraining.MMCTR lacks the query–item relevance labels needed for Stage II; Table 4 reports absolute AUC differences from HiFormer in percentage points.Source: §4.6 · p. 7, Table 4 · p. 7

4.7 WITHIN-PRODUCT CREATIVE EVALUATION

  • +0.75% relative NSCTR improvement exceeded COINS’s +0.27%, indicating stronger within-product creative discrimination beyond ID memorization.The evaluation compares creatives sharing the same product ID.Source: §4.7 · pp. 7–8

4.8 ONLINE A/B TESTING

  • +0.85% GMV and +0.21% Item Page Views (Clicks) were significant improvements over Taobao’s production baseline in a seven-day online A/B test.The evaluation used Taobao’s main search traffic.Source: §4.8 · p. 8, Table 5 · p. 8

5 RELATED WORK

Related work covers multimodal content representations, discrete Semantic ID interfaces, and relevance-supervised transfer to CTR. FineSID combines structured content tokens with a shared predictor trained for relevance and then personalized CTR.

5 RELATED WORK

  • Visual encoders and fine-grained vision–language models demonstrate content representations and interactions across tasks, while industrial CTR must balance content granularity against feature-interaction cost.Examples include FashionBERT (Gao et al., 2020), FILIP (Yao et al., 2022), and MM-Rec.Source: §5 · p. 8
  • Discrete content interfaces include Visual ID (Yu et al., 2022), RQ-VAE SIDs (Singh et al., 2024), and COURIER (Yang et al., 2024a); FineSID retains spatially indexed, multi-scale visual codes and field-specific text tokens.The tokens’ trainable embeddings interact with conventional features during relevance and CTR learning.Source: §5 · p. 8
  • Prior work transfers relevance supervision through pretraining, coupled modules, or joint objectives; FineSID trains shared SID embeddings, interaction layers, and the prediction backbone before personalized CTR fine-tuning.The cited approaches include SEMINAR (Shen et al., 2024), PRECTR (Chen et al., 2025), PRECTR-V2 (Cao et al., 2026), and joint relevance–engagement ranking (Chen et al., 2026).Source: §5 · p. 8

6 DISCUSSION AND LIMITATIONS

The discussion identifies open questions about semantic tokenization, transfer beyond personalized CTR, and computational cost in retrieval. The authors distinguish negligible measured overhead in late-stage ranking from the remaining challenge of extreme low-latency retrieval.

6 DISCUSSION AND LIMITATIONS

  • The visual tokenizer is pretrained for image reconstruction and frozen during relevance pretraining, so it is not strictly optimized for search semantics.Jointly fine-tuning it with relevance pretraining is proposed as future work that could raise the upper bound for cross-modal matching.Source: §6 · p. 9
  • Generalization beyond personalized CTR post-training, including early-stage retrieval and conversion-rate prediction, remains untested.Integrating FineSID’s representation with LLMs is described as an open direction.Source: §6 · p. 9
  • Thirty multi-scale visual tokens widen the initial projection, while extending the framework to extreme low-latency retrieval over millions of candidates remains an optimization challenge.The authors report negligible added overhead and no significant serving-latency increase in late-stage ranking.Source: §6 · p. 9

7 CONCLUSION

FineSID integrates multimodal content into ID-based CTR prediction through progressive learning, establishing a relevance-informed semantic prior before personalization. Evaluations report gains in product-level CTR prediction and within-product creative discrimination.

7 CONCLUSION

  • FineSID unifies fine-grained Semantic IDs, relevance pretraining, and personalized post-training in a shared ID-based CTR architecture.The decoupled design establishes a relevance-informed semantic prior before adapting to implicit personalized user behaviors.Source: §7 · p. 9
  • Offline evaluations on Taobao and MMCTR, alongside online A/B testing, demonstrate the approach’s effectiveness.The paper reports improved standard product-level CTR prediction and strong within-product creative discrimination.Source: §7 · p. 9
  • FineSID’s within-product creative discrimination indicates that it captures fine-grained content differences beyond conventional ID memorization.The conclusion attributes the gains to retaining fine-grained multi-scale structural details through reconstruction-oriented visual tokenization.Source: §7 · p. 9

REPRODUCIBILITY AND AI ASSISTANCE

The appendix supplies evaluation metric formulations and implementation details for reproducibility. The authors disclose AI assistance and retain responsibility for the experimental designs, analyses, and scientific claims.

REPRODUCIBILITY AND AI ASSISTANCE

  • The appendix provides detailed evaluation metric formulations and implementation details, including network configurations and hyperparameter settings.These materials are provided to support reproducibility.Source: REPRODUCIBILITY AND AI ASSISTANCE · p. 9
  • LLMs assisted with literature discovery, research ideation and execution, drafting, writing polish, reference organization, and equation formatting.The authors report that AI-generated content was reviewed and critically revised by the human authors.Source: REPRODUCIBILITY AND AI ASSISTANCE · p. 9
  • The authors state that experimental designs, data analyses, and ultimate scientific claims remain their responsibility.Source: REPRODUCIBILITY AND AI ASSISTANCE · p. 9

A MEASUREMENT DETAILS

The measurement details define group-level AUC and GAUC, which aggregates AUC over valid groups with contributions weighted by impression count. They distinguish this item-level grouping from NSCTR’s within-product creative ranking.

A MEASUREMENT DETAILS

  • For group g, AUCg is Wg divided by n+_g, where Wg counts correctly ordered positive–negative pairs plus half of tied pairs.Source: A MEASUREMENT DETAILS · pp. 11–12
  • GAUC includes valid groups with n+_g > 0 and weights each group’s contribution by its total impression count ng.Source: A MEASUREMENT DETAILS · p. 12
  • Item GAUC compares exposures across users and contexts for a given item, unlike NSCTR, which ranks creatives within the same item ID.Source: A MEASUREMENT DETAILS · p. 12

B IMPLEMENTATION AND REPRODUCIBILITY DETAILS

Implementation details specify multi-scale visual codes and source-specific text encodings, a full-parameter transfer from relevance pretraining to personalized post-training, and the training configuration.

B IMPLEMENTATION AND REPRODUCIBILITY DETAILS

  • Visual tokenization retains 30 spatially indexed codes per image at resolutions 1 × 1, 2 × 2, 3 × 3, and 4 × 4, with missing codes handled by a validity mask.Text sources such as query and title use distinct BPE and n-gram vocabularies.Source: B IMPLEMENTATION AND REPRODUCIBILITY… · p. 12
  • Stage III retains all 197 dense network states and normalization entries from Stage II, while resetting sparse Adagrad accumulators and initializing a fresh CTR training step.Source: B IMPLEMENTATION AND REPRODUCIBILITY… · p. 12
  • Training uses Adagrad at a fixed learning rate of 0.01 across 64 NVIDIA H20 GPUs, with batch size 4,096 per GPU and 262,144 globally.No learning-rate warmup or decay is used; the visual codebook size is V = 4,096.Source: B IMPLEMENTATION AND REPRODUCIBILITY… · p. 12
Loading 2610.29138v1…