Generative End-to-end Ad Retrieval at Douyin

1 INTRODUCTION

• Under continuous distribution shifts, direct tokenizer updates can destabilize gradients and leave the codebook underutilized, while frozen tokenizers cannot adapt dynamically.The passage identifies the lack of a principled scheme for stable direct gradient optimization as a roadblock to fully differentiable end-to-end learning.Source: §1 · p. 1
• Distinct items can share token sequences, and expanding codebook capacity to mitigate these collisions inevitably exacerbates representation collapse.Conventional reranking adds computational or optimization bottlenecks, while appending unique tokens can compromise generalization.Source: §1 · pp. 1–2
• GEAR combines BasisVQ and prefix-aware BasisRQ with a lightweight, context-conditioned reranker jointly optimized to disambiguate colliding items.BasisVQ re-parameterizes the codebook with a learnable orthogonal basis; the reranker reuses generator hidden states and adds minimal inference overhead.Source: §1 · p. 2
• GEAR is validated through offline evaluations and large-scale online A/B tests, serving hundreds of millions of daily active users in production.The framework supports efficient real-time serving at scale.Source: §1 · p. 2

2 PRELIMINARIES

• Each user is represented by a historical behavior sequence and remaining features, while item features include static and dynamic information.Dynamic item features include categorical identifiers with learnable sparse embeddings, motivating adaptation of item representations during training.Source: §2 · p. 2
• An item tokenizer maps item features to an ordered sequence of discrete tokens, which the generator predicts autoregressively for the target item.The generator conditions on the user's behavior sequence and remaining features, along with the begin-of-sequence token and previously generated tokens.Source: §2 · p. 2
• Vector quantization assigns a continuous item representation to its nearest code vector, and residual quantization extends this process across ordered codebooks.Each quantization layer has a codebook size; the resulting tokens form an ordered sequence.Source: §2 · p. 2

3.1 TOKENIZER

• BasisVQ re-parameterizes codebooks with a learnable orthogonal basis, sharing gradients across codes and rotating the latent space without distorting inactive codes.Applying BasisVQ across residual-quantization layers yields BasisRQ; cached codebooks keep inference complexity identical to vanilla VQ.Source: §3.1 · pp. 3–4
• Prefix-aware BasisRQ applies an affine transformation to each layer’s codebook based on earlier quantization decisions, improving expressiveness with the same asymptotic time complexity as vanilla VQ.Its fast distance computation avoids the memory-heavy sample-aware codebook tensor.Source: §3.1 · pp. 4–5, Figure 3 · p. 5

3.2 GENERATOR

• The generator uses an encoder-decoder architecture: user features form a shared key-value representation, and a Transformer decoder autoregressively generates item tokens.The encoder projects non-sequence features and click and conversion sequences into the shared representation.Source: §3.2 · p. 5

3.3 RERANKER

• A jointly learned reranking head scores items after token generation, distinguishing items that share a sequence with one extra decoder step and lightweight dot-product scoring.Each candidate sequence conditions its own user representation for ranking the colliding items.Source: §3.3 · p. 6

3.4 TRAINING AND SERVING

• GEAR jointly trains the tokenizer, generator, and reranker with a weighted four-loss objective, using progressive warm-up before activating quantization, generation, and reranking losses.Generation uses secpm-reweighted cross-entropy; at inference, beam search produces candidate sequences that are scored by the reranker.Source: §3.4 · pp. 6–7

• Prior VQ approaches freeze tokenizers or use heuristic updates, while SimVQ and GRIT-VQ share gradients but retain code-distortion or complexity limitations.Frozen tokenizers do not adapt to dynamic distributions; SimVQ uses an unconstrained transformation, while GRIT-VQ uses a generalized radius surrogate.Source: §4 · p. 7
• Prefix-aware BasisRQ combines orthogonal transformations for streaming updates with lightweight affine prefix conditioning, without increasing vanilla RQ's asymptotic time complexity.Source: §4 · p. 7
• Earlier generative retrieval work spans hierarchical semantic IDs, scalable architectures, and advertising applications, while stable joint optimization of tokenization and generation remains largely unexplored.TIGER established the paradigm; later work includes OneRec, NEZHA, GR4AD, and GPR.Source: §4 · pp. 7–8
• Prior collision approaches append unique tokens, while jointly trained scoring and dense-representation methods have trade-offs in computational overhead, advertising objectives, or lightweight deployment.Source: §4 · p. 8
• GEAR jointly optimizes the tokenizer, generator, and reranker to enable dynamic tokenizer adaptation and resolve item collisions with minimal computational overhead.Source: §4 · p. 8

• GEAR's deployed architecture comprises streaming training, nearline indexing, and online service, with item-dependent computation decoupled from the critical inference path.Source: §5 · p. 8, Figure 4 · p. 8
• Streaming updates maintain item token and embedding entries; a nearline builder creates a compressed sparse row-like index and refreshes snapshots at minute-level granularity.The index maps sorted token sequences to aligned lists of items and precomputed embeddings; snapshots are distributed with the model artifact.Source: §5 · p. 8
• Online inference uses KV-cached beam search, retrieves indexed items, and reranks them with quantized embeddings before per-sequence truncation.Sorting and truncation prevent a single token sequence from dominating results and promote candidate diversity.Source: §5 · p. 8

6 LARGE-SCALE REAL-WORLD EXPERIMENTS

• Online A/B tests ran for seven days with each treatment group receiving 5% of traffic; the evaluated model used a three-layer prefix-aware BasisRQ tokenizer.Generation beam widths were (256, 256, 512).Source: §6.1 · p. 9, §6 · p. 9

6.1 OVERALL PERFORMANCE

• Removing prefix-aware BasisRQ causes the largest ablation drop; the reranking head resolves collisions, and secpm reweighting provides an additional gain.Source: §6.1 · p. 9, Table 1 · p. 9
• ADSS increases by +0.563% and ADVV by +0.658% over the production baseline in full global deployment.Source: Table 1 · p. 9, §6.1 · p. 9
• Cold-start activation rate improves by +1.46% for new advertisers reaching the predefined traffic threshold within the specified time window.The passage describes improved generalization for new ads without historical interaction data.Source: §6.1 · p. 9

6.2 TOKENIZER ANALYSIS

• The proposed tokenizer achieves the highest utilization, lowest collision rate, and lowest quantization error; the progressive configuration beats the uniform schedule at equal capacity.Source: §6.2 · p. 10, Table 2 · p. 10

• GEAR jointly optimizes the tokenizer, generator, and reranker to address representation collapse and item collisions in large-scale industrial recommenders.Source: §7 · p. 10
• Prefix-aware BasisRQ mitigates representation collapse through orthogonal transformations and increased expressiveness, while a lightweight reranker resolves item collisions.Source: §7 · p. 10
• Online A/B tests on Douyin Ads yield consistent revenue gains; future work will explore more efficient decoding to reduce inference latency and broaden deployment scenarios.Source: §7 · p. 10

• Generative AI tools assisted with proofreading, sentence polishing, and checking equations for consistency and errors.Source: GENERATIVE AI DISCLOSURE STATEMENT · p. 10
• The tools were not used to generate original research content or experimental data.Source: GENERATIVE AI DISCLOSURE STATEMENT · p. 10
• The authors reviewed and verified all edited text and accepted responsibility for the final manuscript.Source: GENERATIVE AI DISCLOSURE STATEMENT · p. 10

• The experiment uses 128,000 points from 512 dispersed Gaussian clusters in 8 dimensions, with code vectors initialized near the origin, far from the data manifold.Evaluated methods share the same initialization, optimizer, and loss function.Source: A BASISVQ VS SIMVQ ON SYTHETIC DATA · p. 13
• A single global t-SNE model and identical subplot axes are used to visualize the data and codebooks consistently.Source: A BASISVQ VS SIMVQ ON SYTHETIC DATA · p. 13
• BasisVQ resurrects initially dead codes and uniformly covers the data manifold with a higher convergence rate, while SimVQ fails to explore it fully.SimVQ activates more codes, but its codebook parameters are frozen.Source: A BASISVQ VS SIMVQ ON SYTHETIC DATA · p. 13, Figure 5 · p. 13

• The sample-specific transformed codebook is ˜C(n) = γ(n) ⊙ ˆC + β(n), and the target is the L2 distance matrix D ∈ R^(b×Nm).Source: B DETAILED DERIVATION OF THE FAST… · p. 14
• Subtracting the sample-specific bias defines ˜x(n) = x(n) − β(n), avoiding materialization of a dense tensor with shape (b, Nm, d).Source: B DETAILED DERIVATION OF THE FAST… · p. 14
• The distance computation combines input norms, scaled codebook norms, and a cross term using matrix multiplications in O(bNmd) time, matching vanilla VQ asymptotically.This avoids the prohibitive memory overhead of 3D tensors.Source: B DETAILED DERIVATION OF THE FAST… · p. 14

Tokenizer

• The tokenizer uses 96-dimensional two-tower embeddings and a three-layer Prefix-aware BasisRQ with codebook sizes (2048, 512, 128) and code vector dimension 128.The decreasing capacities align with diminishing residual energy and yield superior performance under the same total codebook capacity.Source: C IMPLEMENTATION DETAILS · p. 14

Generator

• The generator groups non-sequential user features into 28 tokens and uses click and conversion sequence lengths of 200 and 100, respectively.Its two-layer decoder has six 64-dimensional attention heads per layer and a Gated FFN intermediate dimension of 384.Source: C IMPLEMENTATION DETAILS · p. 14

Reranker

• The reranking user representation is set to dimension 96.Source: C IMPLEMENTATION DETAILS · p. 14

Losses and Training

• Training uses loss weights λLTR, λRQ, λGen, and λRerank of 1, 1, 10, and 1, respectively, with generation-step beam widths of 256, 256, and 512 at inference.The LambdaLoss scaling factor σ is 1 for both terms, and the RQ loss hyperparameter η is 0.1.Source: C IMPLEMENTATION DETAILS · p. 15