Source-linked AI summary
Query as Anchor: Scenario-Adaptive User Representation via Large Language Model
Jiahao Yuan, Yike Xu, Jinyong Wen, Baokun Wang, Ziyi Gao, Xiaotong Lin, Yun Liu, Xing Fu, Yu Cheng, Yongchao Liu, Weiqiang Wang, Zhongle Xie
TL;DR
Existing user representation methods struggle to adapt static embeddings to diverse scenarios and to reconcile heterogeneous behavioral logs with language-centric LLM pretraining. The paper introduces Query-as-Anchor, combining semantic pretraining, hierarchical query-conditioned encoding, and soft-prompt adaptation; across 10 benchmarks and 2 online A/B tests, it reports SOTA performance and practical effectiveness.
Problem
Static user representations struggle with scenario adaptability, cross-task generalization, and heterogeneous behavioral data that diverges from language-centric LLM pretraining.
Method
Q-Anchor combines the UserU pretraining corpus, hierarchical coarse-to-fine query-conditioned encoding, and soft prompts for task-adaptive user embeddings.
Results
Across 10 real-world benchmarks and 2 online A/B tests, Q-Anchor achieves SOTA performance and validates practical effectiveness.
Takeaways & Limitations
The framework supports reusable scenario-adaptive embeddings and efficient multi-scenario inference without retraining the backbone model.
Takeaways & Limitations
The reported system operates within the Alipay ecosystem and follows its data-governance, privacy, and security policies.
Abstract
from arXiv · showhide
Industrial-scale user representation learning requires balancing robust universality with acute task-sensitivity. However, existing paradigms primarily yield static, task-agnostic embeddings that struggle to reconcile the divergent requirements of downstream scenarios within unified vector spaces. Furthermore, heterogeneous multi-source data introduces inherent noise and modality conflicts, degrading representation. We propose Query-as-Anchor, a framework shifting user modeling from static encoding to dynamic, query-aware synthesis. To empower Large Language Models (LLMs) with deep user understanding, we first construct UserU, an industrial-scale pre-training dataset that aligns multi-modal behavioral sequences with user understanding semantics, and our Q-Anchor Embedding architecture integrates hierarchical coarse-to-fine encoders into dual-tower LLMs via joint contrastive-autoregressive optimization for query-aware user representation. To bridge the gap between general pre-training and specialized business logic, we further introduce Cluster-based Soft Prompt Tuning to enforce discriminative latent structures, effectively aligning model attention with scenario-specific modalities. For deployment, anchoring queries at sequence termini enables KV-cache-accelerated inference with negligible incremental latency. Evaluations on 10 Alipay industrial benchmarks show consistent SOTA performance, strong scalability, and efficient deployment. Large-scale online A/B testing in Alipay's production system across two real-world scenarios further validates its practical effectiveness. Our code is prepared for public release and will be available at: https://github.com/JhCircle/Q-Anchor.
1 Introduction
Existing user representation methods struggle to combine cross-task transfer with scenario-specific adaptability, while heterogeneous behavior data creates a modality and semantic gap for LLM-based modeling. Query-as-Anchor addresses these challenges through industrial-scale semantic pretraining and query-conditioned scenario adaptation.
- Static user embeddings lack flexibility across downstream scenarios, forcing multiple task-specific models and increasing deployment and maintenance costs.
- LLM-based user encoders face a modality and semantic gap because behavioral logs are sparse, symbolic, and heterogeneous rather than language-centric.
- Query-as-Anchor separates behavior encoding from scenario objectives to generate scenario-adaptive embeddings by re-anchoring the same user profile across contexts.
- UserU aligns heterogeneous behaviors with user-understanding semantics through future-behavior prediction and QA-based supervision.
- Soft-prompt tuning and KV-cache-aware acceleration support efficient scenario specialization and low-latency inference without retraining the backbone.
2 Related Work
Prior work applies language models to behavioral and multimodal user data, but static embeddings limit adaptation to changing contexts. Query-as-Anchor instead uses natural-language query modulation to produce adaptive, domain-specific embeddings from one model.
- Language-model user encoders integrate textual profiles, behavioral sequences, and structured attributes into unified embeddings.
- Earlier behavioral language-model methods use masked prediction or contrastive objectives to capture temporal dependencies but rely on static user embeddings.
- Figure 1 contrasts transferable but fixed general embeddings with Query-as-Anchor modulation for adaptive, domain-specific embeddings via natural-language instructions.
- Synthetic-data research addresses the lack of large-scale datasets designed specifically for user-embedding pretraining through augmentation and pseudo-labeling approaches.
3 Methodology
The methodology combines the UserU pretraining dataset with hierarchical user encoding, query-anchored dual-tower training, and cluster-based soft prompt tuning for scenario-adaptive representations.
- UserU Pretraining Dataset: UserU combines future behavior prediction with synthesized query-answer supervision to align heterogeneous user behaviors with user-understanding semantics.The behavior dataset captures temporal regularities and near-future actions, while the synthetic dataset targets general-purpose user-query-answer alignment.
- Hierarchical Coarse-to-fine User Encoder: The hierarchical encoder projects modality-specific event sequences, preserves fine-grained action features, aggregates modality trends, and derives a global user representation.It processes billing, Mini Program, navigation, app, search, and tabular modalities through event, modality, and user-level representations.
- Q-Anchor Pretraining Architecture: Hierarchical tokens and a trailing natural-language query form query-anchored user embeddings that selectively distill intent-relevant features from the behavioral profile.The architecture uses an anchor tower for hierarchical user tokens and the task-specific query, paired with a semantic tower that encodes answers.
- Q-Anchor Pretraining Architecture: Joint contrastive and next-token prediction objectives make query-anchored embeddings discriminative while retaining semantic information for answer reconstruction.The contrastive loss aligns positive user-answer pairs, whereas next-token prediction adds token-level generative grounding.
- Soft Prompt Tuning: Cluster-based soft prompt tuning freezes the backbone and coarse-to-fine encoder while learning task controllers and class prototypes for discriminative downstream structures.The prototypical contrastive loss pulls prompt-conditioned embeddings toward their category centers and away from irrelevant clusters.
4 Experiments
The experiments evaluate Q-Anchor against general text embeddings and user representation models on 10 Alipay binary classification tasks, using AUC and KS to measure discriminative performance and decision-boundary separation.
- Models and Implementation: All baselines and ablations use identical training conditions, including 50k fine-tuning steps and a global batch size of 2,048.
- Baselines and Tasks: The study compares Q-Anchor with general text embeddings, contrastive user representation methods, and the LLM-based FOUND model.Larger Q-Anchor variants are also evaluated for scalability.
- Evaluation Metrics: Evaluation uses linear probing on 10 Alipay binary classification tasks spanning the user cognition system.
- Evaluation Metrics: AUC measures discriminative performance, while KS measures critical decision-boundary separation.
5 Main Results
Q-Anchor achieves strong, stable performance across 10 Alipay scenarios and benefits from data scaling, prompt tuning, modality structure, contrastive alignment, and pretraining. Online deployments also improve cash-reserve outreach and credit-risk scoring outcomes.
- Overall Performance: Q-Anchor (Prompt Tuned) achieves average AUC 0.8225 and KS 0.5267 across all 10 benchmarks.
- Overall Performance: +0.0737 (+9.84%) in AUC and +0.1462 (+38.4%) in KS versus Llama-Embed-Nemotron-8B.
- Cross-Domain Generalization: The same encoder generalizes across Engagement, Risk, and Marketing without task-specific architectures.
- Scalability: Pretraining data scaling improves Avg. AUC from 0.8029 to 0.8105 and Avg. KS from 0.4895 to 0.5044, whereas model scaling is non-monotonic.
- Scalability: Prompt tuning improves Avg. AUC from 0.8146 to 0.8225 and Avg. KS from 0.5140 to 0.5267 by six prompt tokens, after which performance saturates.
- Interpretability: Prompt tuning raises attention to Bill by +26.0% for Takeout Interest and to SPM by +6.4% for Ant Forest.
- Interpretability: Prompt-tuned representations form clearer scenario-consistent groupings than the universal space in t-SNE and PCA visualizations.
- Ablation Study: Removing contrastive learning reduces Avg. AUC from 0.8104 to 0.7667, while removing pretraining reduces Average AUC from 0.8225 to 0.7781.
6 Conclusion
The paper presents Q-Anchor Embedding as a unified framework combining industrial-scale pretraining, query-conditioned hierarchical encoding, and soft-prompt semantic re-anchoring. Across benchmarks and online tests, it reports strong, transferable, interpretable, and low-cost user representations.
- Contributions: UserU couples future behavior supervision with reflection-verified LLM-generated user QA to provide temporal and semantic priors.
- Contributions: Query-as-Anchor uses hierarchical coarse-to-fine encoding and query-conditioned alignment to distill multi-source behavior into task-adaptive embeddings.
- Contributions: Lightweight soft prompts re-anchor modalities to downstream decision boundaries.
- Conclusion: Across 10 benchmarks and 2 online A/B tests, Q-Anchor reports SOTA AUC and KS with interpretable, scenario-adaptive, low-cost, transferable representations.
Ethical Considerations
The experiments comply with Alipay’s data-governance, privacy, and security policies through encryption, access controls, and audit logging.
- Data Governance: All experiments use encrypted data and embeddings, strict access control, and audit logging to prevent unauthorized access or linkage.
A Data Example of UserU Dataset
UserU is illustrated through synthetic toy examples from two core subsets, showing behavioral prediction and behavior-to-semantic reasoning while preserving privacy.
- The examples come from UserU’s Behavior-based Interaction Dataset (Dfuture) and Synthetic Query-Answer Dataset (Duqa).These subsets illustrate the dataset’s data modality and alignment tasks.
- The cases are synthetic demonstrations and do not represent raw records from any specific individual.
B Details of Downstream Benchmarks
The downstream evaluation uses 10 real-world Alipay binary classification tasks grouped into three domains, with labels based on target events during task-specific prediction windows.
- The test set contains 10 real-world binary classification tasks grouped into three domains.
- Each benchmark is a user-level binary classification task with a task-specific prediction window and target-event label.A user is labeled positive when triggering at least one target event during that window.
- Each task contains approximately 50w test data, with scenario-level label definitions used for privacy and compliance.
- The illustrative UserU figures cover behavior-to-behavior prediction and behavior-to-semantic reasoning.
C.1 Experimental Results on Alipay
Across 10 industrial scenarios, Q-Anchor achieves state-of-the-art KS performance, while ablations identify contrastive learning and pretraining as especially important components.
- Q-Anchor achieves state-of-the-art KS performance across all 10 industrial scenarios.KS measures discriminative ranking of positive versus negative user behaviors.
- Removing contrastive learning lowers Avg. KS from 0.5044 to 0.4215, including a Brand decrease from 0.4527 to 0.2169.
- Skipping pretraining reduces performance by 11.16% even when subsequent prompt tuning is applied.
- Structural tokens, margin-mask filtering, and NTP produce smaller but systematic KS decreases when removed.
C.2 Scalability of Q-Anchor Embedding (Base)
Q-Anchor benefits from more pretraining data, but model-size scaling is non-monotonic; lightweight prompt tuning reaches its best reported performance with six tokens and longer optimization.
- Pretraining data scale: 0.8105 Avg. AUC and 0.5044 Avg. KS are reached at 50k pretraining steps, up from 0.8029 and 0.4895 at 10k steps.The corresponding data scale increases from 20.5M to 102.4M samples with batch size 2048.
- Model scale: 0.8105/0.5044 Avg. AUC/KS is achieved by the 0.5B backbone, while 1.5B and 3B models provide no consistent gains.Larger models slightly regress on several scenarios, including Brand and Promo.
- Model scale: Average gradient magnitude falls from 0.082 at 0.5B to 0.035 at 1.5B and 0.028 at 3B.Maximum gradient magnitude also decreases from 0.824 to 0.231 and 0.164, respectively.
- Prompt-token scale: 0.8225 Avg. AUC and 0.5267 Avg. KS are achieved with six learnable prompt tokens, after rapid gains from one to six tokens and later saturation.
- Learning-step scale: 500 prompt-tuning steps yield 0.8225 average AUC and 0.5267 average KS, compared with 0.8159 and 0.5141 at 100 steps.
- Representation geometry: Prompt-tuned embeddings show tighter intra-class clustering and clearer positive-negative separation across User Engagement, Risk, and Marketing Sensitivity.The effect is especially pronounced in Brand and Money scenarios and aligns with reported AUC and KS improvements.
D Pre-Deployment Offline Performance on Downstream Business
Q-Anchor embeddings outperform production SOTA on two disparate financial tasks, while scenario-specific prompting further improves performance with negligible incremental cost. Deployment combines reusable prefix encoding with lightweight scenario queries and incremental updates, but larger backbones show performance stagnation relative to 0.5B.
- Offline Performance: Q-Anchor Base outperforms Business SOTA across all metrics and delivers a 2.71% absolute lift in IVR KS over handcrafted features.The result is reported without scenario-specific adaptation.
- Offline Performance: 0.3016 KS (+6.05% over SOTA) is achieved in IVR, while delinquency prediction reaches 0.1700 KS (+2.01% over SOTA) with Prompt Tuned Q-Anchor.
- Efficient Adaptation: Prompt tuning reuses the expensive user-prefix KV-cache and optimizes only a few learnable vectors at the query suffix.This supports scenario steering with negligible incremental cost.
- Efficient Serving: Prefix KV-cache sharing encodes each user prefix once, then processes scenario queries sequentially using only short query suffixes.The deployment amortizes prefix encoding across risk, marketing, and engagement scenarios.
- Scaling Limitation: Larger 1.5B–3B backbones exhibit gradient attenuation and performance stagnation compared with the 0.5B model, making 0.5B the primary deployment configuration.The study identifies this as a scaling paradox in user embeddings.