Source-linked AI summary
SAGA: Structure-Attended Generative Action Embedding Model that encodes Multi-Surface User Action Sequences
Tsz Fung Pang, Po Jen Chen, Nimish Ronghe, Farhad Farahani, Bo Zhang
TL;DR
Existing sequential recommenders are often built separately for different surfaces, limiting reusable representations across heterogeneous behavioral domains. SAGA encodes multi-surface action sequences with per-field structure, and its embeddings consistently outperform alternative variants across downstream tasks, supporting a single embedding model across customer touchpoints.
Problem
Surface-specific recommenders lack a reusable backbone for sharing behavioral signals across heterogeneous downstream tasks and touchpoints.
Method
SAGA encodes multi-surface user action sequences using per-field tokenization, field-level attention, and generative training objectives to produce unified user representations.
Results
SAGA-generated representations consistently outperform alternative variants across a range of downstream tasks, with round-robin per-field tokenization showing the greatest improvements.
Takeaways & Limitations
The results suggest that financial service organizations can build one embedding model across heterogeneous customer touchpoints instead of developing surface-specific models.
Takeaways & Limitations
The study did not evaluate the architecture’s potential for surface-specific generative recommendations in downstream tasks.
Abstract
from arXiv · showhide
Prior embedding models for sequential recommendation typically operate within a homogeneous action space, limiting their ability to capture cross-surface behavioral signals spanning distinct behavioral domains. We present SAGA, a generative action embedding model that encodes multi-surface user interaction sequences across a Financial Service organization's ecosystems, from checkout, peer-to-peer (P2P) transactions, in-app engagement, email to account actions, into a unified user representation for downstream recommendation tasks. Central to SAGA is a per-field tokenization schema that decomposes each action event into multiple field-level tokens (e.g. product, interaction, surface), enabling field-level attention and per-field training objectives that fused single-token approaches cannot support. Through an offline ablation study on loss formulation, tokenization granularity and training data scope, we isolate the contribution of each design choice. A downstream model integrated with SAGA-generated user embeddings delivers the strongest overall click and conversion lift across diverse downstream touchpoints, compared to all ablated and alternative architectures.
1 Introduction
SAGA addresses the limits of siloed and homogeneous sequential recommendation by encoding heterogeneous, multi-surface action sequences into a reusable user representation. Its round-robin tokenization and dual-head objective support cross-surface pre-training and transferable downstream performance.
- Siloed recommender systems duplicate feature engineering and produce specialized models without a reusable backbone across downstream tasks.
- Existing sequential recommendation foundation models generally learn within a single platform or homogeneous action space, leaving heterogeneous cross-surface learning underexplored.
- SAGA encodes historical actions across checkout, P2P, app engagement, push notifications, and email into a universal user representation for multiple downstream tasks.
- Round-robin tokenization enables field-level attention across heterogeneous action attributes, while the dual-head objective combines autoregressive field prediction with event-boundary contrastive learning.
- SAGA validates that multi-surface pre-training makes diverse data sources complementary rather than conflicting and produces representations transferable across downstream tasks.
2 Related Work
SAGA extends generative recommendation and financial foundation-model research to heterogeneous, multi-surface customer activity. Its unified event schema and per-field round-robin autoregressive tokenization preserve surface-specific structure that fused representations discard.
- Overview: SAGA’s related-work contributions span generative recommendation, financial foundation models, and action event tokenization for reusable downstream recommendation representations.The study examines representation learning from heterogeneous financial and digital customer activities.
- Generative recommendation: SAGA targets heterogeneous multi-surface behavior, unlike prior models designed for relatively homogeneous action spaces or single platforms.The modeled activities include checkout, P2P payments, app sessions, web visits, emails, and push notifications.
- Financial foundation models: SAGA jointly models financial and non-financial customer activities, broadening the scope of prior financial foundation models focused on banking histories, payment trajectories, or transaction-level representations.This supports evaluation of how diverse digital and communication signals complement financial behavior in downstream recommendation.
- Action event tokenization: SAGA decomposes each event into seven fields, contrasting with PinFM’s one token and HSTU’s two-token event representations.The seven fields comprise six event attributes plus a separator token (K=7).
- Action event tokenization: SAGA applies per-field round-robin tokenization autoregressively, preserving surface-specific information for reusable embeddings across heterogeneous surfaces.The approach extends encoder tokenization into autoregressive modeling and is presented as the first such application in recommender systems.
3 Methodology
SAGA constructs chronologically ordered multi-surface action sequences using a unified six-attribute event schema, then expands each event into seven round-robin tokens for granular modeling. It trains a frozen decoder-only transformer with autoregressive and event-based contrastive objectives to produce downstream user embeddings.
- Embedding Extraction: The frozen model’s final-layer hidden state after the full user sequence serves as the user embedding input to downstream models.Fine-tuning is outside the paper’s scope, and downstream models are trained and evaluated using embeddings generated from frozen SAGA weights.
- Data Construction: The dataset aggregates user actions from checkout, P2P transactions, app engagement, push notifications, emails, and account management into a proprietary financial-service corpus.Actions from different surfaces are featurized into an identical structured schema and ordered chronologically.
- Unified Event Schema: Each event is standardized into six attributes: surface, parent product, product, action intent, interaction, and merchant code categorization.The attributes are produced through a predetermined lookup approach.
- Mechanism: SAGA tokenizes every event into K=7 round-robin fields: BOS, surface, parent product, product, action, interaction, and merchant code.The seven-field structure extends HSTU’s two-field interleaving and contrasts with Pi’s single fused event embedding.
- Motivations: The seven-field design enables per-field attention, constrained next-event generation, and surface-conditioned user embeddings for generative recommendation.The transformer can attend separately to attributes from prior events and condition generation on a selected surface.
- Autoregressive Predictions: SAGA uses a dual-head architecture combining next-token loss, multi-token loss, and an event-based InfoNCE contrastive loss.The multi-token objective follows round-robin fields across future events, while the contrastive head aggregates the prior event’s seven hidden states and applies stop-gradient to prevent collapse.
4 Offline Experiments
Offline ablations show that SAGA’s dual-head objectives, per-field tokenization, and all-surface training data produce the most transferable representations for downstream click and conversion ranking. These gains arise from complementary field- and event-level signals, richer tokenization, and broader behavioral coverage.
- Experimental setup: The offline study evaluates training objectives, tokenization granularity, and training-data scope using downstream in-App product-recommendation ranking.Downstream rankers use identical ESMM/DCN-v2 architectures and hyperparameters, with production features and optional frozen embeddings.
- Training objectives: Dual-head SAGA outperforms single-head variants overall, transferring the most click and conversion gains across downstream touchpoints.Contrastive-only improves conversion more on touchpoint-A but degrades most on touchpoint-C, whereas dual-head SAGA contributes the least noise interference there.
- Training objectives: The dual-head objectives provide complementary signals: LM heads learn field-level transition dynamics, while the contrastive head learns distinguishable event-level representations.Neither objective alone captures both granularities.
- Tokenization granularity: SAGA’s per-field tokenization achieves the highest gains across most touchpoints, and under equal token budgets matches or exceeds fused-key and interleaved designs with 7× fewer events.In Set B, fused-key’s touchpoint-A advantage disappears, while fused-key degrades severely on touchpoint-C.
- Training data composition: All-surface training produces the greatest click and conversion lifts across touchpoints, while adding transaction data alone generally deteriorates performance.The passage characterizes transaction data as low-frequency and strong-signal, versus engagement data as high-frequency and weak-signal.
5 Online A/B Testing Results
SAGA was deployed in production A/B tests on touchpoints B and C, with treatment recommendations using a downstream model and frozen SAGA embeddings. It improved both reported metrics on touchpoint B and Metric B on touchpoint C, while touchpoint C’s Metric A decline was not statistically significant.
- A/B test design: Production treatment recommendations used a downstream model with SAGA’s frozen embedding, while control used the prior deep learning ranker without a pre-trained embedding.The control relied on engagement-only sequences plus user-level aggregated data from other surfaces.
- Touchpoint-B results: +12.75% improvement in Metric A and +8.8% lift in Metric B were statistically significant on touchpoint-B.Table 6 reports both gains for the SAGA treatment group.
- Touchpoint-C results: -1.56% drop in Metric A on touchpoint-C was not yet statistically significant, while Metric B improved by +11.2%.The touchpoint-C Metric A result aligned with the offline noise-sensitivity analysis in Section 4.
6 Future Work
Future work will fine-tune model variants on downstream tasks, evaluate surface-specific generative recommendations enabled by round-robin tokenization, and validate scaling laws across model sizes and backbones.
- Future work: Fine-tune model variants by unfreezing model weights to test different architectures’ capacity on downstream tasks.The current study leaves downstream-task fine-tuning out of scope.
- Future work: Evaluate round-robin tokenization for constrained, surface-specific generative recommendations in downstream tasks.This capability is a theoretical architectural benefit that was not evaluated.
- Future work: Validate scaling laws for structure-attended generative action models by varying model size and backbone.The study fixed both model size and backbone.
7 Conclusion
The conclusion identifies round-robin per-field tokenization (K=7), combined contrastive and next-token objectives, and cross-surface learning as key contributors to SAGA’s downstream performance. It further suggests that financial service organizations can build one embedding model across heterogeneous customer touchpoints instead of surface-specific models.
- 7 Conclusion: Round-robin per-field tokenization (K=7) enabled structure-attended action sequences with the greatest performance improvements across downstream tasks.Given the same token budget, richer per-field tokenization generated more informative representations.
- 7 Conclusion: Contrastive training and next-token prediction are mutually complementary, and their combination is more effective than either objective alone.The conclusion presents the combined objectives as a stronger training formulation for the embedding model.
- 7 Conclusion: Cross-surface learning benefits user representation compared with training on single-domain data.The study supports learning across heterogeneous customer touchpoints within one embedding model rather than developing surface-specific models.