Sona Technical Report

• SONA more than doubles Argus on primary Active Users.Each deployment adds its gain on top of gains retained from previous deployments.Source: Figure 1 · p. 1
• Bars show incremental A/B uplift in percent, with each deployment adding its gain to those retained from earlier deployments.Panels use independent scales.Source: Figure 1 · p. 1
• V0, V1, V2, and Argus are successive transformer-based recommendation generations deployed before SONA.Source: Figure 1 · p. 1

• My Vibe on smart speakers is a pure-recommendation test setting, with playback starting without a specified artist, genre, or mood.The surface is also among Yandex Music’s largest by listening time.Source: §1 · p. 3
• Music recommendation must balance familiar tracks that sustain time spent with discovery of new favorites that pays off over a longer horizon.Listening is largely passive, so feedback arrives as plays, skips, and occasional likes; repeated listens are signal rather than redundancy.Source: §1 · p. 3
• SONA generates Semantic ID tuples, expands them to catalog tracks, and ranks candidates through a shared encoder evaluated once per request.It uses logged event fields and learned Semantic IDs rather than hand-crafted features.Source: §1 · p. 3
• The Teacher Ranker is trained with next-item pre-training followed by multi-head ranking fine-tuning, and supplies distillation targets without hand-engineered features.The introduction describes pre-training as enabling a teacher-only transformer to replace a feature-rich production ranking model.Source: §1 · p. 3
• The online-training loop delivers fresh model weights every 10 minutes, with event-to-production update latency of 45 minutes at median and 60 minutes at the 99th percentile.Events are aggregated into sessions over a 15-minute window before flowing into real-time processing and the training queue.Source: §1 · p. 3
• +4.53% Active Users, +6.30% Total Listening Time, and +11.42% Likes in SONA’s final online experiment.The Active Users uplift was 2.35× the increment previously delivered by Argus, the strongest model previously deployed on the surface.Source: §1 · p. 3

2.1 Design Philosophy

• The encoder is evaluated once per request, and its user representation is reused for both generation and ranking.The design adopts Gryphon as its unified generation-and-ranking architecture.Source: §2.1 · p. 4
• SONA and its teacher use logged event fields and learned Semantic IDs rather than hand-engineered features.The design rationale is to explore how far this approach reaches in music recommendation.Source: §2.1 · p. 4
• Next-token prediction supplies dense sequence targets, while ranking feedback is sparse; teacher pre-training followed by ranking fine-tuning is the initial recipe.Ablations find pre-training crucial for teacher quality.Source: §2.1 · p. 4
• A teacher trained autoregressively on a year of engagement logs compresses long history into its weights, then distills knowledge into the Ranking Module trained on a shorter date range.Source: §2.1 · p. 4
• Rollout Distillation uses frozen-teacher scores on current decoder rollouts to train the Ranking Module, aligning its signal with the decoder’s candidate distribution.Because the modules share the encoder, the signal also shapes the representation conditioning both tasks.Source: §2.1 · p. 4

2.2 Overall Framework

• At serving, the decoder generates Semantic ID tuples, a catalog index maps them to tracks, and the Ranking Module scores and orders candidates using shared encoder states.Source: §2.2 · p. 5, Figure 2.1 · p. 5
• During training, next-token and teacher-distillation losses both back-propagate through the shared encoder; the teacher and tokenizer are absent from final serving.Final serving uses the encoder, decoder, deterministic SID-to-item mapping, and Ranking Module.Source: §2.2 · p. 5

3.1 Semantic Tokenizer

• Offline tokenization maps each track to a three-code Semantic ID tuple using multimodal content features, collaborative refinement, and residual quantization.The three codebooks each contain 32,000 entries.Source: §3.1 · pp. 6, 8
• Collaborative refinement trains on related track pairs with a contrastive loss and aligns each item embedding to its content features (λalign = 0.1).The pair sources include co-served items and co-listening tracks; the latter are filtered to high-NPMI pairs from the same artist.Source: §3.1 · pp. 6–7

3.2 User History Representation

• Each history is a chronological sequence of logged engagement events, embedded through item identity, request context, and feedback fields without hand-engineered features.Track duration and completion ratio are bucketed and embedded rather than used as raw scalars.Source: §3.2 · p. 8

3.3 Encoder

• History Compression preserves all N positions in shared memory while applying its seven-layer deep stack only to the recent block.With N = 8192 and nr = 2048, it preserves most downstream performance of a full transformer over the same history at about half the inference cost.Source: §3.3 · pp. 9–10

3.4 Decoder

• The decoder generates each track’s Semantic ID tuple autoregressively from BOS, conditioned on the encoded history.Constrained beam search masks invalid prefixes, then expands completed tuples into their catalog-track collision classes.Source: §3.4 · p. 10

3.5 Ranking Module

• The Ranking Module scores beam-search candidates against the same encoder memory, then orders them using a fixed weighted combination of its teacher-distilled head scores.Reusing the encoded history means scoring adds no second encoder pass.Source: §3.5 · p. 10

4.1 Dataset Design

• Request-level samples pair each served request’s preceding chronological history with its impressions and subsequent feedback; training sweeps forward in time.Empty-history samples are dropped, and each sample conditions only on events before its request.Source: §4.1 · p. 11
• Positive impressions become packed Semantic ID targets for next-token prediction, while logged impressions and decoder rollouts supply ranking-distillation candidates.Multiple positive targets share one encoder pass, and catalog-missing impressions are masked from the loss.Source: §4.1 · p. 11, §4.2 · pp. 11–12

4.2 Training Framework

• Next-token prediction and distillation toward the frozen Teacher Ranker jointly update the shared encoder alongside the decoder and Ranking Module.The decoder learns from positive impressions; the Ranking Module matches the teacher’s per-candidate engagement scores using mean absolute error.Source: §4.2 · p. 11, Figure 4.1 · p. 12
• At serving, a fixed weighted combiner collapses the distilled engagement-head scores to reorder beam candidates, but this combined score is not a training loss.The joint loss sums next-token prediction and rollout and impression distillation terms.Source: §4.2 · p. 12
• Initial training uses day-level chronological chunks with requests shuffled within each day, then online training consumes requests in arrival order at a constant terminal learning rate.AdamW, weight decay, gradient clipping, and FSDP are used during training.Source: §4.2 · p. 12

5.1 Architecture

• The Teacher Ranker combines a deep encoder over typed user-event histories with a cross-attention candidate scorer and engagement heads, without hand-engineered features.Its causal encoder provides a timeline-specific user state, while the deep stack attends to the full history without History Compression.Source: §5.1 · p. 13

5.2 Dataset Design

• Timeline-level samples let one causal encoder pass supervise every target event, using a year of target events rather than the several weeks in Sona’s request-level dataset.Pre-training samples contain timelines; fine-tuning and refresh samples also include served impressions and graded engagement feedback.Source: §5.2 · pp. 13–14

5.3 Training Framework

• Next-item-prediction pre-training learns the causal history encoder from every timeline position before its prediction head is discarded.The objective is InfoNCE over in-batch and uniformly sampled tail negatives, with LogQ correction for in-batch popularity bias.Source: §5.3 · p. 14
• Ranking fine-tuning warm-starts the encoder and trains a candidate scorer to order temporally adjacent served impressions by graded engagement, with auxiliary binary-feedback heads.The primary pairwise objective compares different engagement grades; fine-tuning stays on recommendation-request impressions and attends to history preceding each target.Source: §5.3 · p. 14
• After fine-tuning, the teacher is rolled forward and refreshed daily on newly logged sessions from the recommendation surface it supervises.The paper identifies next-item pre-training’s contribution as an ablation target.Source: §5.3 · p. 14

6.1 Training Optimizations

• Training co-locates Sona’s generation, ranking, and frozen-teacher evaluation on shared GPU workers, while overlapping input preparation and hash computation with GPU execution.Mixed precision, FSDP, FlashAttention, torch.compile, QK normalization, and gradient clipping support execution and numerical stability.Source: §6.1 · pp. 14–15

6.2 Online Training Infrastructure

• The online loop aggregates events into sessions, joins them with inference-time context, applies traffic gates, and feeds joint-objective training; updated weights reach serving every 10 minutes.Event-to-serving update latency is 45 minutes at the median and 60 minutes at the 99th percentile; the Teacher Ranker is refreshed into the trainer every 24 hours.Source: §6.2 · pp. 15–16, Figure 6.1 · p. 16

6.3 Inference Optimizations

• Inference runs the history encoder once and reuses its states for Semantic ID decoding and candidate ranking, keeping intermediate candidates on the GPU.A CPU batcher packs variable-length histories for CUDA-graph replay; beam search uses radix-based top-k selection and catalog-prefix constraints.Source: §6.3 · p. 17
• Decoder attention uses tile sizes tuned for large-beam decoding’s many one-token queries and short key–value sequences, while maintaining a KV cache across steps.The described key–value sequences contain at most three positions.Source: §6.3 · p. 17

7.1 Evaluation Protocol and Metrics

• Offline evaluation separates target-track retrieval, candidate-conditioned ranking fidelity, and engagement-weighted pair ordering; all three metrics are higher-is-better.When a configuration lacks a Ranking Module, decoder likelihood supplies its ordering for Teacher Recall.Source: §7.1 · pp. 17–18

7.2 Candidate Generation

• Candidate-generation ablations select 3 × 32k codebooks with refined Qwen2.5-Omni; larger models lower train loss, and longer one-pass windows improve point estimates without an observed plateau.The capacity sweep measures train NTP loss at matched target exposure; data-volume runs make one chronological pass over longer log windows.Source: §7.2 · pp. 18–19

7.3 Teacher Ranker Design Ablations

• Teacher Ranker ablations retain NTP pre-training and six-layer, 8k-event scoring, with longer attended histories consistently improving WPA across scorer depths.The pre-training comparison holds history length and scorer depth fixed; the history and depth sweep selects the six-layer, 8k-event configuration for subsequent work.Source: §7.3 · pp. 19–20, Table 7.4 · p. 20, Table 7.5 · p. 20

7.4 Unified Generation and Ranking

• No tested distillation loss dominates retrieval, ranking fidelity, and engagement ordering, so MAE remains the reference for subsequent ablations.The comparison includes MAE, MSE, Huber, and pairwise KL.Source: §7.4 · p. 21, Table 7.6 · p. 21
• Combining decoder rollouts with logged impressions keeps Teacher Recall near rollouts alone while giving the highest WPA point estimate, so both sources are retained.Impressions alone have the highest retrieval point estimate but much lower Teacher Recall than configurations containing rollouts.Source: §7.4 · p. 21
• Increasing the training rollout beam from 32 to 128 gives the highest Teacher Recall point estimates but quadruples scored candidates, so 32 is retained as the lower-cost reference.The training beam controls candidates scored during training and is distinct from the fixed evaluation beam of 1024.Source: §7.4 · p. 21
• A four-layer Ranking Module raises Teacher Recall further; with that depth fixed, full 8k-event attention leads Target-track Recall and Teacher Recall, while History Compression has the highest WPA point estimate.History Compression applies the deep encoder only to recent events, and this configuration is selected for the final model.Source: §7.4 · p. 21, Table 7.7 · p. 21

7.5 Online A/B Tests

• +4.53% Active Users uplift for SONA was 2.35× Argus’s prior increment, as the full model replaced the recommendation stack in Experiment 5.Experiment 5 ran the full SONA recipe with an 8192-event history and History Compression.Source: Table 7.12 · p. 23, §7.5 · p. 23

8 Conclusion, Limitations, and Future Work

• SONA replaces candidate generation, pre-ranking, and ranking with one end-to-end model and improves all five reported engagement metrics relative to production control.The report also describes the system's limitations and future directions.Source: §8 · p. 24
• SONA has not been deployed on full traffic; a multi-month experiment is planned, and its catalog coverage is lower than the production stack’s.The report says the coverage difference will be investigated.Source: §8 · p. 24
• The approach has been validated on one recommendation surface, and deployment on other surfaces remains future work.The report frames cross-surface validation as necessary for full deployment.Source: §8 · p. 24
• Across the measured capacity and data ranges, point estimates continue to improve at the largest tested settings.Planned scaling directions include larger capacity, sparse mixture-of-experts backbones, sparse and linear attention, and a persistent cross-request KV cache.Source: §8 · p. 24
• How to use test-time scaling effectively remains open; widening beam search trades inference compute for candidate quality.Source: §8 · p. 24
• The report observed no degradation in content exploration, possibly because SONA is trained on production logs, but fresh-content behavior remains for future investigation.Source: §8 · p. 24
• The reported recipe is fully supervised because reinforcement learning has not yet been shown to contribute in a controlled comparison.Controlled evaluation of long-term reward modeling and RLVR-style post-training on user feedback is left to future work.Source: §8 · p. 24

B.1 Semantic Tokenizer

• Appendix B.1 provides a model card for the Semantic Tokenizer.Source: Table B.1 · p. 27

B.2 Teacher Ranker

• The Teacher Ranker’s initial training has two stages.Source: Table B.2 · p. 28
• Only the encoder is reused between pre-training and multi-head ranking fine-tuning.Source: Table B.2 · p. 28
• The resulting Teacher Ranker is refreshed continuously.Source: Table B.2 · p. 28

B.3 Ranking Module

• The Ranking Module model card describes a module distilled from the Teacher Ranker jointly with encoder–decoder backbone training.Source: Table B.3 · p. 28

B.4 Encoder–Decoder Backbone

• Appendix B.4 includes an encoder model card.Source: Table B.4 · p. 28
• Appendix B.4 includes a decoder model card.Source: Table B.5 · p. 28
• Appendix B.4 includes an encoder–decoder backbone optimization card.Source: Table B.6 · p. 29

B.5 Online Training and Model Refresh

• Appendix B.5 provides a configuration table for online training and model refresh.Source: Table B.7 · p. 29

C Notation

• Appendix C provides notation used throughout the report.Source: Table C.1 · p. 30