PaperlayerView exact revision on arXiv

Source-linked AI summary

SAGA: Structure-Attended Generative Action Embedding Model that encodes Multi-Surface User Action Sequences

Tsz Fung Pang, Po Jen Chen, Nimish Ronghe, Farhad Farahani, Bo Zhang

arXiv:2608.15429v1cs.LGcs.IR

TL;DR

Existing sequential recommenders are often built separately for different surfaces, limiting reusable representations across heterogeneous behavioral domains. SAGA encodes multi-surface action sequences with per-field structure, and its embeddings consistently outperform alternative variants across downstream tasks, supporting a single embedding model across customer touchpoints.

  • Problem

    Surface-specific recommenders lack a reusable backbone for sharing behavioral signals across heterogeneous downstream tasks and touchpoints.

    Source: §1 · pp. 2–3

  • Method

    SAGA encodes multi-surface user action sequences using per-field tokenization, field-level attention, and generative training objectives to produce unified user representations.

    Source: §1 · p. 2, §3.2 · p. 5, §3.3 · p. 5

  • Results

    SAGA-generated representations consistently outperform alternative variants across a range of downstream tasks, with round-robin per-field tokenization showing the greatest improvements.

    Source: §1 · p. 3, §7 · p. 8

  • Takeaways & Limitations

    The results suggest that financial service organizations can build one embedding model across heterogeneous customer touchpoints instead of developing surface-specific models.

    Source: §7 · p. 8

    The study did not evaluate the architecture’s potential for surface-specific generative recommendations in downstream tasks.

    Source: §6 · p. 8

Abstract

from arXiv · show

Prior embedding models for sequential recommendation typically operate within a homogeneous action space, limiting their ability to capture cross-surface behavioral signals spanning distinct behavioral domains. We present SAGA, a generative action embedding model that encodes multi-surface user interaction sequences across a Financial Service organization's ecosystems, from checkout, peer-to-peer (P2P) transactions, in-app engagement, email to account actions, into a unified user representation for downstream recommendation tasks. Central to SAGA is a per-field tokenization schema that decomposes each action event into multiple field-level tokens (e.g. product, interaction, surface), enabling field-level attention and per-field training objectives that fused single-token approaches cannot support. Through an offline ablation study on loss formulation, tokenization granularity and training data scope, we isolate the contribution of each design choice. A downstream model integrated with SAGA-generated user embeddings delivers the strongest overall click and conversion lift across diverse downstream touchpoints, compared to all ablated and alternative architectures.

1 Introduction

SAGA addresses the limits of siloed and homogeneous sequential recommendation by encoding heterogeneous, multi-surface action sequences into a reusable user representation. Its round-robin tokenization and dual-head objective support cross-surface pre-training and transferable downstream performance.

  • Siloed recommender systems duplicate feature engineering and produce specialized models without a reusable backbone across downstream tasks.Source: §1 · p. 2
  • Existing sequential recommendation foundation models generally learn within a single platform or homogeneous action space, leaving heterogeneous cross-surface learning underexplored.Source: §1 · p. 2
  • SAGA encodes historical actions across checkout, P2P, app engagement, push notifications, and email into a universal user representation for multiple downstream tasks.Source: §1 · p. 2
  • Round-robin tokenization enables field-level attention across heterogeneous action attributes, while the dual-head objective combines autoregressive field prediction with event-boundary contrastive learning.Source: §1 · p. 3
  • SAGA validates that multi-surface pre-training makes diverse data sources complementary rather than conflicting and produces representations transferable across downstream tasks.Source: §1 · p. 3

2 Related Work

SAGA extends generative recommendation and financial foundation-model research to heterogeneous, multi-surface customer activity. Its unified event schema and per-field round-robin autoregressive tokenization preserve surface-specific structure that fused representations discard.

Overview

  • SAGA’s related-work contributions span generative recommendation, financial foundation models, and action event tokenization for reusable downstream recommendation representations.The study examines representation learning from heterogeneous financial and digital customer activities.Source: §2 · p. 3

Generative recommendation

  • SAGA targets heterogeneous multi-surface behavior, unlike prior models designed for relatively homogeneous action spaces or single platforms.The modeled activities include checkout, P2P payments, app sessions, web visits, emails, and push notifications.Source: §2.1 · p. 3

Financial foundation models

  • SAGA jointly models financial and non-financial customer activities, broadening the scope of prior financial foundation models focused on banking histories, payment trajectories, or transaction-level representations.This supports evaluation of how diverse digital and communication signals complement financial behavior in downstream recommendation.Source: §2.2 · p. 3

Action event tokenization

  • SAGA decomposes each event into seven fields, contrasting with PinFM’s one token and HSTU’s two-token event representations.The seven fields comprise six event attributes plus a separator token (K=7).Source: §2.3 · pp. 3–4
  • SAGA applies per-field round-robin tokenization autoregressively, preserving surface-specific information for reusable embeddings across heterogeneous surfaces.The approach extends encoder tokenization into autoregressive modeling and is presented as the first such application in recommender systems.Source: §2.3 · p. 4

3 Methodology

SAGA constructs chronologically ordered multi-surface action sequences using a unified six-attribute event schema, then expands each event into seven round-robin tokens for granular modeling. It trains a frozen decoder-only transformer with autoregressive and event-based contrastive objectives to produce downstream user embeddings.

Embedding Extraction

  • The frozen model’s final-layer hidden state after the full user sequence serves as the user embedding input to downstream models.Fine-tuning is outside the paper’s scope, and downstream models are trained and evaluated using embeddings generated from frozen SAGA weights.Source: §3 · p. 4, §3.4 · p. 6

Data Construction

  • The dataset aggregates user actions from checkout, P2P transactions, app engagement, push notifications, emails, and account management into a proprietary financial-service corpus.Actions from different surfaces are featurized into an identical structured schema and ordered chronologically.Source: §3.1 · pp. 4–5

Unified Event Schema

  • Each event is standardized into six attributes: surface, parent product, product, action intent, interaction, and merchant code categorization.The attributes are produced through a predetermined lookup approach.Source: §3.1 · p. 5

Mechanism

  • SAGA tokenizes every event into K=7 round-robin fields: BOS, surface, parent product, product, action, interaction, and merchant code.The seven-field structure extends HSTU’s two-field interleaving and contrasts with Pi’s single fused event embedding.Source: §3.2 · p. 5

Motivations

  • The seven-field design enables per-field attention, constrained next-event generation, and surface-conditioned user embeddings for generative recommendation.The transformer can attend separately to attributes from prior events and condition generation on a selected surface.Source: §3.2 · p. 5

Autoregressive Predictions

  • SAGA uses a dual-head architecture combining next-token loss, multi-token loss, and an event-based InfoNCE contrastive loss.The multi-token objective follows round-robin fields across future events, while the contrastive head aggregates the prior event’s seven hidden states and applies stop-gradient to prevent collapse.Source: §3.3 · pp. 5–6

4 Offline Experiments

Offline ablations show that SAGA’s dual-head objectives, per-field tokenization, and all-surface training data produce the most transferable representations for downstream click and conversion ranking. These gains arise from complementary field- and event-level signals, richer tokenization, and broader behavioral coverage.

Experimental setup

  • The offline study evaluates training objectives, tokenization granularity, and training-data scope using downstream in-App product-recommendation ranking.Downstream rankers use identical ESMM/DCN-v2 architectures and hyperparameters, with production features and optional frozen embeddings.Source: §4 · p. 6

Training objectives

  • Dual-head SAGA outperforms single-head variants overall, transferring the most click and conversion gains across downstream touchpoints.Contrastive-only improves conversion more on touchpoint-A but degrades most on touchpoint-C, whereas dual-head SAGA contributes the least noise interference there.Source: §4.1 · pp. 6–7
  • The dual-head objectives provide complementary signals: LM heads learn field-level transition dynamics, while the contrastive head learns distinguishable event-level representations.Neither objective alone captures both granularities.Source: §4.1 · p. 7

Tokenization granularity

  • SAGA’s per-field tokenization achieves the highest gains across most touchpoints, and under equal token budgets matches or exceeds fused-key and interleaved designs with 7× fewer events.In Set B, fused-key’s touchpoint-A advantage disappears, while fused-key degrades severely on touchpoint-C.Source: §4.2 · p. 7

Training data composition

  • All-surface training produces the greatest click and conversion lifts across touchpoints, while adding transaction data alone generally deteriorates performance.The passage characterizes transaction data as low-frequency and strong-signal, versus engagement data as high-frequency and weak-signal.Source: §4.3 · p. 7

5 Online A/B Testing Results

SAGA was deployed in production A/B tests on touchpoints B and C, with treatment recommendations using a downstream model and frozen SAGA embeddings. It improved both reported metrics on touchpoint B and Metric B on touchpoint C, while touchpoint C’s Metric A decline was not statistically significant.

A/B test design

  • Production treatment recommendations used a downstream model with SAGA’s frozen embedding, while control used the prior deep learning ranker without a pre-trained embedding.The control relied on engagement-only sequences plus user-level aggregated data from other surfaces.Source: §5 · p. 8

Touchpoint-B results

  • +12.75% improvement in Metric A and +8.8% lift in Metric B were statistically significant on touchpoint-B.Table 6 reports both gains for the SAGA treatment group.Source: §5 · p. 8

Touchpoint-C results

  • -1.56% drop in Metric A on touchpoint-C was not yet statistically significant, while Metric B improved by +11.2%.The touchpoint-C Metric A result aligned with the offline noise-sensitivity analysis in Section 4.Source: §5 · p. 8

6 Future Work

Future work will fine-tune model variants on downstream tasks, evaluate surface-specific generative recommendations enabled by round-robin tokenization, and validate scaling laws across model sizes and backbones.

Future work

  • Fine-tune model variants by unfreezing model weights to test different architectures’ capacity on downstream tasks.The current study leaves downstream-task fine-tuning out of scope.Source: §6 · p. 8
  • Evaluate round-robin tokenization for constrained, surface-specific generative recommendations in downstream tasks.This capability is a theoretical architectural benefit that was not evaluated.Source: §6 · p. 8
  • Validate scaling laws for structure-attended generative action models by varying model size and backbone.The study fixed both model size and backbone.Source: §6 · p. 8

7 Conclusion

The conclusion identifies round-robin per-field tokenization (K=7), combined contrastive and next-token objectives, and cross-surface learning as key contributors to SAGA’s downstream performance. It further suggests that financial service organizations can build one embedding model across heterogeneous customer touchpoints instead of surface-specific models.

7 Conclusion

  • Round-robin per-field tokenization (K=7) enabled structure-attended action sequences with the greatest performance improvements across downstream tasks.Given the same token budget, richer per-field tokenization generated more informative representations.Source: §7 · p. 8
  • Contrastive training and next-token prediction are mutually complementary, and their combination is more effective than either objective alone.The conclusion presents the combined objectives as a stronger training formulation for the embedding model.Source: §7 · p. 8
  • Cross-surface learning benefits user representation compared with training on single-domain data.The study supports learning across heterogeneous customer touchpoints within one embedding model rather than developing surface-specific models.Source: §7 · p. 8

Related papers

Loading 2608.15429v1…