Source-linked AI summary
How Do Decoder-Only LLMs Perceive Users? Rethinking Attention Masking for User Representation Learning
Jiahao Yuan, Yike Xu, Jinyong Wen, Baokun Wang, Yang Chen, Xiaotong Lin, Wuliang Huang, Ziyi Gao, Xing Fu, Yu Cheng, Weiqiang Wang
TL;DR
The paper studies how attention masking affects decoder-only LLMs used for user representation learning, a setting where causal, hybrid, and bidirectional choices remain insufficiently compared. It evaluates these masks in a unified contrastive framework and introduces Gradient-Guided Soft Masking, which consistently improves training stability and bidirectional representation quality across industrial user-centric tasks.
Problem
The impact of causal, hybrid, and bidirectional attention masking on decoder-only LLM user representations remains underexplored.
Method
The paper compares three masking strategies in a unified contrastive framework and uses Gradient-Guided Soft Masking before a linear scheduler to transition toward bidirectional attention.
Results
Gradient-Guided Soft Masking consistently yields more stable training and higher-quality bidirectional representations than causal, hybrid, and scheduler-only baselines across 9 industrial user-centric tasks.
Takeaways & Limitations
Masking design and the transition dynamics from causal to bidirectional attention are important when adapting decoder-only LLMs for user representation learning.
Abstract
from arXiv · showhide
Decoder-only large language models are increasingly used as behavioral encoders for user representation learning, yet the impact of attention masking on the quality of user embeddings remains underexplored. In this work, we conduct a systematic study of causal, hybrid, and bidirectional attention masks within a unified contrastive learning framework trained on large-scale real-world Alipay data that integrates long-horizon heterogeneous user behaviors. To improve training dynamics when transitioning from causal to bidirectional attention, we propose Gradient-Guided Soft Masking, a gradient-based pre-warmup applied before a linear scheduler that gradually opens future attention during optimization. Evaluated on 9 industrial user cognition benchmarks covering prediction, preference, and marketing sensitivity tasks, our approach consistently yields more stable training and higher-quality bidirectional representations compared with causal, hybrid, and scheduler-only baselines, while remaining compatible with decoder pretraining. Overall, our findings highlight the importance of masking design and training transition in adapting decoder-only LLMs for effective user representation learning. Our code is available at https://github.com/JhCircle/Deepfind-GGSM.
1 Introduction
Decoder-only LLMs offer autoregressive user modeling, but the effect of causal, hybrid, and bidirectional attention masks on user representation quality remains insufficiently studied. This work compares these masking strategies and introduces a gradient-guided transition mechanism for more stable bidirectional adaptation.
- Decoder-only LLMs support autoregressive interaction, unlike bidirectional models that require full context upfront for interactive settings.
- Causal, bidirectional, and hybrid masks provide distinct recipes for adapting decoder-only LLMs to user representation learning.Hybrid masking applies bidirectional attention to a user segment and causal attention to downstream tokens.
- The study systematically examines masking and training dynamics within a unified contrastive framework using real-world Alipay user cognition benchmarks.
- Training transition from causal to bidirectional attention affects optimization stability and representation quality.
- Gradient-Guided Soft Masking provides a gradient-informed pre-warmup for smoother causal-to-bidirectional transition and stronger final representations.The method is evaluated on 9 user-centric classification benchmarks.
2 Related Work
Prior work uses bidirectional models for holistic user embeddings but faces limitations in interactive settings, while decoder-only models enable autoregressive modeling. Existing studies suggest bidirectional masking can improve representation quality, but the broader synthetic-data literature also highlights concerns about proprietary generation pipelines.
- Encoder-based bidirectional models capture contextual dependencies but require full input visibility, limiting streaming or interactive use.
- Decoder-only LLMs provide an autoregressive alternative for user modeling when adapted with contrastive objectives.
- Bidirectional masking yields the highest representational quality, while hybrid masking offers a trade-off with generative compatibility.
- Synthetic user-modeling data is increasingly generated with LLMs because high-quality labeled data remains scarce.
- Proprietary APIs used in synthetic-data pipelines raise concerns about cost, reproducibility, and domain alignment.
3 Training Data
The training data combines rule-based behavioral trajectories with LLM-synthesized query–answer alignments built from real-world Alipay interactions. The synthesis pipeline probes alignment difficulty, extracts rules from challenging positive pairs, and scales and rewrites the resulting data for embedding training.
- LLM-synthesized query–answer alignments represent user intent and language understanding through user profiles, generated queries, and answers.
- User profiles integrate multimodal interactions over 90 days, including PayBill, Mini Program, SPM, search, and tabular features.
- Behavioral trajectories pair users’ past interaction sequences with representative future behavior targets from subsequent activity.The left tower uses the prior three months, while the target is sampled from an aggregated subsequent one-month window.
- The synthesis pipeline initializes a seed pool and retrieves the 10 most relevant scenarios for each user before generating query–answer pairs.A calibration set contains 1,000 user–query-answer pairs.
- Alignment difficulty is measured with cosine similarity, and samples above a threshold are retained as challenging pairs.The retained set is denoted D_hard and contains pairs with S_d ≥ T_filter.
- Rules extracted from hard-to-align positive pairs are integrated into prompts for further query–answer synthesis.
- The optimized synthesis process scales the dataset and applies posterior rewriting to improve clarity, context alignment, and semantic consistency.
4 LLMs as Encoders: Training Recipe
The training recipe combines a dual-tower decoder-only LLM architecture with contrastive alignment and Gradient-Guided Soft Masking to transition from causal to bidirectional attention.
- Training Architecture: User and answer towers share the decoder-only LLM backbone but encode independently after modality-specific inputs are projected into the LLM embedding space.This preserves modality-aware representation learning while maintaining compatibility with the LLM’s token semantics.
- Gradient-Guided Soft Masking: Gradient-Guided Soft Masking uses gradient norms during warmup to assign future-attention visibility, then freezes those weights before linearly opening attention toward full bidirectionality.At inference, the model uses a fully bidirectional mask.
- Training Objective: The contrastive objective aligns normalized user and answer embeddings by increasing positive-pair similarity relative to in-batch negatives.The temperature controls similarity smoothness, while the normalization factor aggregates positive and negative similarities.
- Training Objective: A mask factor suppresses sufficiently similar negatives to mitigate false negatives in contrastive training.Same-side negative samples further improve representation distinctiveness and answer-embedding separability.
5 Experiments
Experiments fine-tune a compact decoder-only Qwen backbone with heterogeneous behavioral embeddings and compare multiple attention-mask recipes against user-representation baselines on Alipay tasks.
- Models and Implementation: Qwen2.5-0.5B-Instruct serves as the decoder-only LLM backbone, receiving concatenated modality-specific embeddings from dedicated gte-base-zh encoders.Qwen3-30B-A3B is used in the data-training pipeline for efficiency.
- Baselines and Tasks: The experiments compare causal attention with hybrid strategies that open future attention using gradient-guided masking, an MLP, or a global mechanism.The causal recipe retains the original causal mask under contrastive learning.
- Baselines and Tasks: The evaluation covers 9 real-world Alipay user scenarios using binary classification, alongside U-MLP One4all, MSDP, CPC, and FOUND baselines.All models are evaluated with consistent training hyperparameters.
- Data Information: Table 1 organizes user-pretraining and test-benchmark data information, including the number of tests per task.The table provides the task-level test-count layout for the benchmark data.
- Evaluation Metrics: AUC measures discriminative performance through linear probing on 9 annotated binary classification tasks from Alipay’s user cognition system.AUC is reported as the evaluation metric for user representations.
6 Main Results
Across nine downstream tasks, GG-SM delivers strong and consistent user-embedding performance while gradient-guided attention evolution improves the transition toward bidirectional modeling. The results cover masking-strategy comparisons, domain-specific benchmarks, and training convergence.
- Parameter Efficiency and Domain-Specific Alignment: 0.7745 average AUC: GG-SM outperforms Llama-embed-nemotron (0.7357) and KaLM-Embedding (0.7156) across downstream tasks.It also achieves Transit: 0.6978 and Power: 0.9689.
- From Local Contrast to Contextual Priors: GG-SM provides more consistent gains than FOUND (0.7690) and InstructUE (0.7728), especially for tasks requiring global contextual transfer.Traditional baselines can excel on local-feature tasks such as Achiev. (0.6256) but struggle with preferences requiring broader context.
- Efficacy of Gradient-Guided Attention Evolution: Gradient-guided soft masking outperforms naive bidirectional and scheduler-based methods by using gradient norms to prioritize informative tokens during early adaptation.The comparison links the transition path to embedding quality and sharper separation of user interests on Behavior Preference tasks.
- Evaluation Overview: Figure 3 compares training-loss convergence for GG-SM and the scheduler, while Figure 2 reports average AUC across masking strategies and embedding baselines.Table 2 provides task-level AUC results across User Prediction, Behavior Preference, and Marketing Sensitivity.
- Domain Robustness and Transferability: GG-SM maintains a stable performance advantage across User Prediction, Behavior Preference, and Marketing Sensitivity, achieving peak AUC in Marketing Sensitivity.The result indicates robustness across distinct user-understanding domains.
7 Conclusion
The paper systematically studies attention masking for decoder-only LLM user representation learning and identifies transition dynamics as important alongside the final mask. Gradient-Guided Soft Masking improves optimization behavior and bidirectional representations while retaining decoder-pretraining compatibility.
- The study compares causal, hybrid, and bidirectional masks under a unified contrastive framework using real-world Alipay data and nine industrial user-centric tasks.
- Gradient-Guided Soft Masking serves as a pre-warmup before a linear scheduler that gradually transitions from causal to bidirectional modeling.
- The transition path affects training stability and embedding quality, and the proposed method yields stronger bidirectional representations while remaining compatible with decoder pretraining.
A.1 Standardized Input Template
The standardized input template represents each user with heterogeneous multimodal records and optional instructions, using explicit delimiters and a special <USER> token for representation extraction.
- Each user instance is formatted as a heterogeneous sequence of multimodal records collected over the past 90 days.
- Explicit semantic boundary tokens such as <bill>...</bill> and <minipro>...</minipro> preserve modality structure for modality-aware encoding.
- An optional user instruction qi may be appended after the user profile in the complete model input.
- The special token <USER> aggregates preceding multimodal information and any provided instruction into a unified user representation.
A.2 User Embedding Extraction
The decoder-only backbone maps each input sequence to final-layer token hidden states, using the special <USER> token’s state as the unified user embedding. L2 normalization prepares the representation for contrastive learning.
- The decoder-only causal LLM produces final-layer hidden states for every token in an input sequence.Each hidden state h_t belongs to the t-th token and lies in R^d.
- The unified user embedding is the hidden state at the <USER> token position.
- L2 normalization is applied to stabilize the resulting user representation before contrastive alignment.
A.3 Answer Embedding Extraction
Answer embeddings are extracted with the same decoder-only backbone and aligned contrastively, while masking recipes vary from strict causality to progressively opened bidirectional attention. The proposed transition uses gradient-derived soft masks before interpolation to full bidirectionality for stable convergence.
- Answer Embedding Extraction: Each answer is independently encoded with the same LLM backbone, appending <EOS> before extracting its answer embedding.
- Answer Embedding Extraction: Normalized user and answer embeddings are aligned through an InfoNCE contrastive objective.
- Causal Masking: Causal masking restricts each token to attending only to itself and preceding tokens, preserving left-to-right information flow and generative compatibility.
- Hybrid Masking: Hybrid masking relaxes causality over user-history tokens while retaining causal constraints for future tokens through gradient-, MLP-, or global-query-based variants.
- Bidirectional Masking: Gradient-Guided Soft Masking warms up with gradient-derived soft masks, then linearly interpolates toward full bidirectionality to enable stable convergence.