PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards a Densing Law for User Representation Learning at Billion-Scale Capacity

Bin Dou, Junru Zhang, Zhaoyi Yuan, Wuliang Huang, Letian Gong, Baokun Wang, Huan Li, Yu Cheng, Weiqiang Wang

arXiv:2608.23392v1cs.IRcs.AI

TL;DR

Industrial user representation learning faces diminishing returns when scaling raw behavioral data, users, and model capacity. The paper introduces the Behavioral Densing Law and adaptive tokenization to characterize and allocate sufficient representation capacity. Experiments on Alipay production data report saturation in conventional scaling and improved scaling behavior from densified representations, while generalization beyond the studied modality remains unvalidated.

  • Problem

    Industrial scaling of users, behavioral history, and model capacity encounters diminishing returns from redundant behavioral information, while the required tokenization capacity at different data scales is not established quantitatively.

  • Method

    The paper formulates the Behavioral Densing Law and uses RQ-VAE tokenization plus adaptive variable-length allocation to estimate and distribute sufficient discrete representation capacity.

  • Results

    Conventional scaling saturates on Alipay production data, whereas densified representations shift scaling toward a more favorable performance-cost frontier and can let smaller encoders with shorter inputs outperform larger raw-history models.

  • Takeaways & Limitations

    The proposed law provides practical guidance for selecting tokenization capacity as behavioral data scale changes, and ALGN improves capacity allocation within this framework.

  • Takeaways & Limitations

    The experiments focus on a single data modality, so generalization to other behavioral modalities remains to be validated.

Abstract

from arXiv · show

User representation learning in real-world industrial scenarios is commonly scaled by increasing user amount, behavioral sequence length and model size. However, existing methods face two challenges: (i) Bottleneck for raw data scaling at billion-scale capacity, as performance exhibit diminishing performance gains with larger-scale raw text user behavioral input, which can be mitigated by tokenization. (ii) Lack of quantitative analysis of how tokenization configurations should scale with data size. In this report, we propose User Behavioral Densing Law for characterizing the quantitative relationship between data scale and the minimum sufficient tokenization capacity. Firstly, we conduct a pilot study on raw & tokenized scaling comparison on billion-scale Alipay dataset, revealing the raw data scaling bottleneck and the sustained gains enabled by tokenization. To derive the scaling pattern governing the minimum sufficient tokenization configuration at different data scales, theoretical analysis and systematic experiments are employed to summarize the quantitative scaling pattern. We find an approximately linear relationship between the logarithms of minimum sufficient tokenization capacity and input data size measured by tokens, and the scaling slope varies systematically with the tokenization method and data source, reflecting differences in representation-space redundancy and intra-source uniqueness. Guided by the proposed law, we further develop ALGN, an adaptive variable-length tokenization method that improves capacity allocation. Extensive experiments across diverse data sources, tokenization methods, and downstream tasks demonstrate the generalizability and reliability of the User Behavioral Densing Law, providing practical guidance for tokenization configuration selection in large-scale user representation learning. Moreover, ALGN outperforms existing baselines.

1 Introduction

The introduction identifies a raw behavioral scaling wall: larger user populations, longer histories, and bigger models can add redundant information without improving downstream quality. It proposes behavioral densing and adaptive tokenization to preserve task-relevant signal in compact representations and guide capacity allocation.

  • The paper asks whether representation quality can improve without simply processing more raw behavioral events.
  • The raw behavioral scaling wall describes diminishing downstream returns, including lower pre-training loss without improved downstream accuracy.
  • Scaling encoder parameters from 0.2B to 0.4B mostly fits redundant behavioral details, leaving downstream representation accuracy almost unchanged.
  • Behavioral densing compresses long histories while preserving task-relevant distinctions and suppressing redundancy.
  • RQ-VAE tokenization maps repetitive histories into compact discrete codes, while variable-length allocation gives more capacity to high-entropy segments.
  • Matched token and compute budgets show tokenized representations overcoming the raw scaling wall with consistent gains and better encoder utilization on Alipay PayBill data.
  • The Behavioral Densing Law estimates required tokenization capacity from behavioral data scale and lightweight statistics after tokenizer-method calibration.
  • Adaptive Length Gated Network allocates variable residual depth according to quantization residual and expression uncertainty.

2 Related Work

Related work covers raw behavioral scaling, tokenized behavioral representations, and adaptive token usage. The reviewed methods motivate studying redundancy, representation capacity, and input-dependent token allocation together.

  • Raw behavioral data scaling: Related studies examine how behavioral data, model capacity, and computation affect downstream utility, while long-history methods improve extended-sequence utilization.
  • Raw behavioral data scaling: Additional behavioral records may be repetitive or weakly informative, producing diminishing marginal gains in raw-data scaling studies.
  • Tokenized behavioral data: Behavioral tokenization converts raw user data into compact discrete representations, including reusable units learned through VQ-VAE and residual quantization through RQ-VAE.
  • Capacity configuration and adaptive tokenization: Codebook size, residual depth, and token length determine tokenized-representation capacity, but existing methods commonly use manually selected configurations.
  • Capacity configuration and adaptive tokenization: Adaptive quantization lets different inputs activate different numbers of quantization levels, extending beyond fixed capacity configurations.

3 Preliminary

The paper establishes a behavior-text contrastive pretraining pipeline and evaluates user representations across classification, text-based retrieval, and user-to-user retrieval tasks.

  • Notation: User behavioral data comprise chronological, multi-source records from PayBill, SPM, and MiniProgram activity across users and time windows.
  • Model Pre-training Framework: Pretraining splits each behavioral sequence into historical and future segments, encoding the history with a Transformer and the future with a LoRA-tuned LLM embedding model.
  • Model Pre-training Framework: A contrastive Info-NCE objective aligns Transformer-generated behavioral embeddings with embeddings generated from textual descriptions of future behavior.
  • Downstream Evaluation Protocol: Evaluation uses three industrial user-representation tasks: classification, text-based retrieval, and user-to-user retrieval across diverse application scenarios.
  • Downstream Evaluation Protocol: Classification is assessed with AUC, KS, and Accuracy, while both retrieval tasks use AUC, Precision, and Recall.

4 Scaling Law Analysis: Raw Data Scaling Wall

Scaling raw users, behavioral history, or encoder size produces strong gains only before redundancy dominates. Beyond the saturation regime, additional data and parameters mainly increase cost without improving downstream representation quality.

  • Scaling diagnosis: Across user population, temporal horizon, and model capacity, downstream probing performance shows rapidly diminishing returns as raw scale increases.
  • User population scaling: 0.03B users marks an approximate saturation point: growth from 0.01B improves all three metrics, whereas scaling to 0.1B yields only marginal gains.
  • Temporal horizon scaling: 60 days marks an approximate temporal saturation point: extending history from 30 to 60 days helps, but 60 to 120 days adds limited improvement despite doubling input.
  • Model capacity scaling: 0.2B parameters is an approximate model-capacity threshold: scaling from 0.05B to 0.2B improves accuracy, while 0.4B adds almost no downstream gain.
  • Quality dissociation: Lower pretraining loss can coexist with unchanged downstream accuracy because extra capacity fits redundant behavioral details rather than generalizable user features.
  • Overall scaling pattern: The steepest improvements occur within N ≤0.03B, D ≤60 days, and M ≤0.2B; beyond these thresholds, raw scaling incurs substantial cost with negligible downstream gains.
  • Density-oriented scaling: Residual quantization operationalizes density-oriented scaling by converting long, repetitive raw histories into compact discrete behavioral tokens under matched data and compute budgets.

5 Densing Gains from Tokenization

Raw behavioral scaling eventually reaches diminishing returns because added histories contain substantial redundancy, while tokenization increases effective information density and preserves gains at larger scales. The section presents RQ-VAE as a practical tokenizer that compresses long histories into compact discrete sequences for downstream user encoding.

  • Motivation: Raw behavioral scaling yields diminishing downstream returns once repetitive and weakly informative events dominate user histories.The limitation appears across sequence length, user population, and model capacity.
  • Tokenization method: RQ-VAE compresses multi-source behavioral embeddings into a fixed budget of compact discrete tokens, decoupling effective input length from the raw temporal horizon.A local aggregation step produces H ≪ L_n latent vectors before residual quantization.
  • Tokenization method: Residual quantization provides coarse-to-fine codes that capture dominant behavioral semantics first and refine predictive residual variations later.The method is designed to preserve finer behavioral distinctions than a single codebook.
  • Scaling comparison: Around D ≈64 days, tokenized representations begin outperforming raw sequences, with the advantage increasing at longer observation horizons.At shorter horizons, raw sequences remain competitive because redundancy is more limited.
  • Scaling comparison: Around N ≈1.2 × 10^7 users, tokenized representations consistently outperform raw baselines as raw population scaling approaches saturation.At smaller user scales, the two representations perform similarly.
  • Implication: Tokenization gains become positive and larger in the raw-modeling diminishing-return regime by suppressing repetitive details while preserving task-relevant distinctions.The benefit is therefore not merely shorter input sequences but higher effective behavioral information density.
  • Limitation: Fixed-length tokenization is suboptimal because behavioral information is heterogeneous across history segments.This motivates allocating token capacity adaptively according to behavioral complexity.

6 The Behavioral Densing Law

The Behavioral Densing Law formulates tokenization as a Pareto optimization between downstream utility and representation cost, then characterizes how minimal sufficient capacity changes with behavioral scale. Experiments across methods, data sources, and tasks find approximately logarithmic linear scaling, with slopes shaped by tokenization-method redundancy and intra-source uniqueness.

  • Formulation: The Behavioral Densing Law formalizes densing as a Pareto optimization over downstream utility and representation cost.RQ-VAE is one concrete solver for estimating the resulting capacity trajectory on large-scale behavior data.
  • Formulation: A larger cost sensitivity λ favors cheaper representations, whereas a smaller λ permits higher capacity when it produces sufficient downstream gain.The objective targets the best utility-cost tradeoff rather than maximum compression or tokenizer size.
  • Capacity selection: The law selects minimal sufficient capacity by choosing the smallest configuration that reaches a near-optimal utility bound under stochastic training noise.This avoids larger configurations whose additional capacity yields negligible downstream improvement.
  • Scale law: Required optimal capacity grows as a power-law function of behavioral complexity under a diminishing-return utility frontier.User population and temporal horizon are treated as controllable behavioral-scale dimensions in this study.
  • Empirical validation: Minimal sufficient tokenization capacity increases monotonically with user population or temporal horizon, while exceeding it sharply reduces cost efficiency with negligible performance benefit.The fitted trajectory represents the empirical solution path of the Behavioral Densing Law.
  • Empirical validation: Across input sources and downstream tasks, ln(C*) and ln(s) exhibit an approximately linear relationship.The validation covers RQ-VAE, VQ-VAE, and SARQ across Paybill, SPM, and Miniprogram data sources.
  • Slope analysis: VQ-VAE has a higher fitted slope than RQ-VAE, while SARQ has the lowest, consistent with differences in representation-space redundancy.RQ-VAE enables compositional code reuse, whereas SARQ alleviates low-probability or invalid code combinations.
  • Slope analysis: Within a tokenization method, slopes approximately track squared intra-source uniqueness across data sources.Observed slope ratios are around 1 : 1.15 : 0.92, while corresponding uniqueness ratios are around 1 : 1.07 : 0.96.

7 New Method: Adaptive-Length-Gating-Based RQ-VAE Quantization

ALGN turns the Behavioral Densing Law into instance-level capacity allocation by assigning residual-code depth according to behavioral information and cost. It improves efficiency and downstream quality over fixed- and variable-length baselines.

  • Adaptive capacity allocation: ALGN replaces uniform code depth with variable-length tokenization, continuing quantization only when expected marginal utility exceeds representation cost.The method decides at each quantization level whether another residual code is needed.
  • Adaptive capacity allocation: Residual information and code uncertainty serve as signals for deciding whether behavioral representations need additional quantization levels.Larger residual norms indicate insufficient reconstruction, while greater uncertainty indicates more complex or ambiguous behavior.
  • Adaptive capacity allocation: High-information periods receive deeper semantic IDs, whereas routine or redundant periods stop earlier under ALGN.This local allocation rule operationalizes minimal sufficient capacity for individual behavioral periods.
  • Results: 1.07%/2.03%/0.36% AUC/KS/Acc gains and 13.23% capacity savings distinguish ALGN from the best baseline SARQ.The comparison uses fixed input settings with 180-day sequences and 0.1B users.
  • Results: 2.00%/5.21%/1.14% AUC/KS/Acc improvements accompany capacity reduction from 86.91% to 63.47% versus the heuristic variable-length baseline.The result supports adaptive variable SID length control over heuristic allocation.
  • Ablation results: Removing either adaptive signal worsens efficiency and performance, while the proposed regularization achieves the strongest overall efficiency–performance trade-off.Without uncertainty, capacity usage rises to 77.83%; removing regularization raises it to 73.42%.
  • Scaling effect: ALGN reduces the Densing Law scaling slope to ∼0.59, attributed to lower representation-space redundancy.This evaluates the method’s effect on the scaling relationship itself.

8 Conclusion & Limitation

The paper argues that conventional scaling reaches saturation because behavioral tokens carry limited predictive information, and proposes density scaling through compact representations. It reports practical benefits while acknowledging that broader behavioral modalities and public benchmarks remain unvalidated.

  • Conclusion: Alipay production experiments show saturation across user population, temporal horizon, and model capacity, including unchanged downstream accuracy beyond 0.2B parameters.Increasing users beyond 0.03B or temporal windows beyond 60 days produces rapidly diminishing returns, while training loss can continue decreasing.
  • Conclusion: RQ-VAE tokenization and matched-budget evaluation operationalize density scaling, with compact discrete codes carrying more task-relevant signal than raw sequences.The densified representation shifts the scaling curve toward a more favorable exponent.
  • Practical implications: The resulting tokens are presented as reusable across user targeting, profile prediction, and CTR estimation, with saturation thresholds informing production decisions.The proposed applications include data retention, training-budget, and architecture planning.
  • Practical implications: Density scaling is positioned as an alternative to collecting more data, seeking better representations from existing data at lower scaling cost.The paper connects this motivation to computational and environmental scrutiny of indiscriminate data scaling.
  • Limitations: The study focuses on a single data modality, so generalization of the Densing Law to modalities such as video consumption remains unvalidated.The authors are also evaluating robustness on public benchmark datasets beyond one platform.
  • Future directions: The paper frames information density as a design objective alongside model capacity and data volume, motivating research on measurable, optimizable signal per token.It identifies adaptive tokenization, multimodal density estimation, and theoretical foundations as future directions.

9 Appendix

The appendix reports scaling behavior across training loss, alignment accuracy, and heterogeneous data sources. Longer behavioral histories improve alignment metrics, while compression configurations transfer across sources with largely preserved AUC.

  • Scaling behavior: D ≥90-day contrastive-loss curves nearly overlap, indicating that additional historical days contribute minimal new optimization signal.All configurations converge, but final-loss reductions diminish sharply as the timespan increases.
  • Scaling behavior: 8.28 to 8.15 training loss accompanies behavioral timespan growth from 30 to 180 days.The loss converges to progressively lower values as the input timespan increases.
  • Scaling behavior: 0.29 to 0.42 T2U accuracy and 0.35 to 0.50 U2T accuracy show positive scaling with longer behavioral inputs.Both directions of contrastive representation alignment improve across the reported timespan range.
  • Data-source generalization: Figure 13 evaluates data-source generalization of the compression strategy across heterogeneous sources.The accompanying result compares transfer performance with source-consistent settings.
  • Data-source generalization: 95.0% and 97.5% of source-consistent AUC are preserved when the Bill-derived compression configuration transfers to SPM and Miniprogram, respectively.The result supports compression robustness under cross-source distribution shifts.
Loading 2608.23392v1…