PaperlayerView exact revision on arXiv

Source-linked AI summary

ScalarLens: Numerical Embeddings with Stable Coordinates and Contextual Responses for CTR Prediction

Heng Yao, Tianying Liu, Yulou Shu, Yong He, Chuan Yuan, Kaibin Qiu, Guowei Chen, Jiayu Zhao, Siyun Hou

arXiv:2609.29182v1cs.IR

TL;DR

Numerical encoders assign a scalar one representation even when its predictive evidence depends on sample context, while external normalization adds synchronization work across production stages. ScalarLens separates a scalar-only coordinate from a bounded contextual response, changing only numerical embeddings. It ranks first in 25 of 27 combinations of datasets and backbones on original values.

  • Problem

    Numerical encoders assign a scalar the same token across samples, while external normalization requires transformations and statistics to stay synchronized across training and serving.

    Source: §1 · p. 1

  • Method

    ScalarLens combines a monotone coordinate determined only by the focal scalar with a separate bounded, low-rank response conditioned on the sample.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    ScalaRLens ranks first in 25 of 27 combinations of datasets and backbones while accepting original values.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The paper frames numerical embedding as separating an invariant coordinate belonging to a value from an adaptive response belonging to a value in context.

    Source: §6 · p. 8

    ScalaRLens is not a universally dominant predictor; categorical context is reliable, while numerical context is auxiliary on real data and simpler designs remain competitive in some settings.

    Source: §5 · p. 8

Abstract

from arXiv · show

Numerical embeddings for click-through rate (CTR) prediction are built on a convenient but restrictive premise: a scalar has one representation. This premise conflates where a value lies with what it means for the current sample. On the Criteo validation split, the same numerical interval carries residual click evidence with opposite signs across categorical and numerical contexts, even after additive main effects are removed. Production pipelines compound this mismatch because externally normalized features require transformations and statistics to remain synchronized between training and serving. We introduce ScalarLens, a numerical embedding that preserves what a value is while adapting how it should be interpreted. A monotone local mesh constructs a stable coordinate from the focal scalar alone; bounded low-rank dynamics then produce a contextual response without moving that coordinate or replacing categorical tokens and the CTR backbone. In a 1,539-run primary evaluation covering 19 representations, three datasets, nine backbones, and three seeds, ScalarLens ranks first in 25 of 27 settings on original numerical scales and second in the remaining two. Matched ablations show that scale correction, additional local capacity, and generic conditioning do not reproduce the gain. A controlled study further recovers categorical, numerical, and mixed response mechanisms under context shift while the focal coordinate remains exactly invariant. A complete rerun under shared standardization retains significant advantages over DEER, DAES, and NaryDis, showing that the result is not explained by tolerance to raw scales alone. ScalarLens therefore recasts numerical embedding as a measurement problem: coordinates belong to values, while predictive responses belong to values in context.

1 Introduction

The introduction frames numerical embeddings as conflating a scalar’s coordinate with its context-dependent predictive meaning, alongside deployment costs from external normalization. ScalarLens separates a scalar-only coordinate from a bounded contextual response while retaining categorical tokens and the CTR backbone.

1 Introduction

  • Existing numerical methods give a value the same token before feature interaction, conflating its coordinate with its predictive meaning.The introduction describes existing discretized, local, spectral, hierarchical, and distributional geometries.Source: §1 · p. 1
  • External normalization adds synchronization work across training, backfills, and serving, beyond fitting moments or materializing transformed columns.The paper frames learning from original-scale values as a deployment requirement, while noting it does not remove schema checks, clipping, or drift monitoring.Source: §1 · p. 1
  • For Criteo I7’s (2, 3] interval, residual CTR is −3.2, +3.5, and +5.8 points across C17 groups, and +6.8, +1.9, and −7.2 across I11 ranges.The residuals subtract empirical Bayes additive estimates of target-bin and context-group main effects.Source: §1 · p. 1, Figure 1 · p. 2
  • ScalarLens anchors a monotone mesh to the scalar and adds a bounded low-rank contextual response, changing numerical embeddings without replacing categorical tokens or the CTR backbone.Its construction makes the coordinate stable while the response adapts to the sample.Source: §1 · p. 2
  • ScalaRLens ranks first in 25 of 27 dataset–backbone combinations; matched checks show tolerance to input scale alone does not explain the result.The evaluation covers 1,539 runs.Source: §1 · p. 2

2 Related Work

Related work spans numerical representations, general contextual refinement, and conditional numerical encoding. The paper distinguishes these approaches by where context acts relative to a scalar coordinate and response.

2 Related Work

  • Prior work spans numerical representation, general contextual refinement, and conditional numerical encoding; the paper distinguishes these lines by where context acts.Table 1 positions the approaches by their treatment of coordinates and responses.Source: §2 · p. 2, Table 1 · p. 3

2.1 Numerical Representation for CTR

  • Numerical encoders vary in their bases and geometries, but still assign each scalar the same token across samples.Examples include soft assignments, local interpolation, frequency components, splines, learned meshes, numeral-system scales, and distributional anchors.Source: §2.1 · p. 2

2.2 Contextual Feature Representation

  • General contextual refiners adapt latent features, potentially rewriting categorical and numerical fields without preserving an explicit ordered scalar coordinate.The matched ContextFiLM ablation conditions numerical outputs on selected categorical fields while fixing output shape and consumer.Source: §2.2 · p. 2

2.3 Conditional Numerical Encoding

  • DAES conditions interpolation weights over a maintained quantile coordinate; ScalaRLens instead conditions a response downstream of a scalar-only coordinate.ScalaRLens accepts original values without external normalization or maintained streaming quantiles, and does not require a specialized consumer.Source: §2.3 · pp. 2–3

3 ScalaRLens: Stable Coordinates and Contextual Responses

ScalarLens separates a scalar-only stable coordinate from a bounded response conditioned on sample context, while leaving categorical embeddings and the CTR backbone unchanged. It combines an ordered local mesh with fixed-depth shared low-rank dynamics.

3.1 One Numerical Embedding, Two Coupled Mechanisms

  • The design separates a scalar-only coordinate from a context-conditioned response, using a model-reduction analogy with shared low-rank communication and bounded fixed-depth updates.The analogy supplies an inductive bias, not a claim that CTR data obey a physical law.Source: §3.1 · p. 3

3.2 Stable Numerical Coordinates

  • Equal focal values retain identical coordinates across contexts; ordered local interpolation clips values outside the training range, while source validation and drift monitoring remain necessary.Training ranges are stored with the checkpoint, avoiding materialized normalized copies or reproducing a mean-and-variance transformation online.Source: §3.2 · pp. 3–4

3.3 Contextual Response Dynamics

  • RMS-normalized field tokens yield field-specific drives and gates that update a bounded response through shared low-rank communication, without changing the focal coordinate.The response remains bounded at every update, independently of input scale.Source: §3.3 · p. 4

3.4 Numerical Readout that Preserves the Coordinate

  • The readout retains the focal coordinate alongside gate and transient states, returns only numerical tokens, and leaves categorical tokens to the unchanged backbone.Keeping the coordinate explicit means the response state need not reconstruct location.Source: §3.1 · p. 3, §3.4 · p. 4

3.5 Complexity and Implementation Boundary

  • Interpolation costs O(BNK) and response dynamics cost O(BTMr); the implementation uses batched projections and needs no sample loop, matrix inverse, or convergence test.The remaining heads scale linearly with the number of fields.Source: §3.5 · p. 4

4 Experiments

ScalarLens ranks first in 25 of 27 original-scale settings and second in the other two; experiments also probe ablations, controlled response recovery, sensitivity, and computational cost.

4.1 Datasets, Backbones, and Shared Protocols

  • The shared evaluation spans three datasets, nine backbones, 19 representations, and three seeds, retaining original numerical units and the same categorical path and backbone.Each matrix contains 1,539 runs; validation AUC selects checkpoints, and test AUC is reported for selected checkpoints.Source: §4 · p. 4, §4.1 · pp. 4–5

4.2 Comparison with Numerical Embedding Methods

  • ScalarLens ranks first in 25 of 27 settings and second in two, with mean AUC gains of 0.0070/0.0037/0.0012 over each dataset’s strongest competitor.It wins eight of nine backbones on each AutoML dataset and all nine on Criteo; DEER leads by 0.0005 on AutoML-A/AutoInt, while B-spline ties at displayed precision on AutoML-E/AutoInt.Source: Table 3 · p. 5, §4.2 · pp. 5–6
  • Under shared z-score standardization, ScalaRLens remains ahead of DEER, DAES, and NaryDis by +0.0031, +0.0047, and +0.0103 mean AUC, with all paired comparisons significant after Holm correction.The audit removes all 60 cells below 0.55 AUC; it is a secondary audit, not the primary deployment result.Source: §4.2 · p. 6

4.3 Ablation Study

  • Removing response costs 0.0005–0.0091 AUC in all six cells; adding categorical context to numerical-only improves all 18 paired seeds, while other numerical context is auxiliary.The focal coordinate is preserved across source variants, identifying categorical context as reliable and numerical context as auxiliary rather than uniformly necessary.Source: §4.3.1 · p. 6, §4.3.3 · p. 6
  • NoNorm, StopGrad, and T=1 lose 0.0037/0.0025/0.0011 mean AUC; DEER+Norm and LocalMLP lose 0.0043 each, while ContextFiLM loses 0.0011.FrozenMessage is mixed (−0.0002 average), so message recomputation is not a defining claim; the matched controls test stabilization, local capacity, and affine conditioning.Source: §4.3.2 · p. 6, §4.3.3 · p. 6

4.4 Controlled Mechanism Recovery and Context Shift

  • On shifted categorical, numerical, and mixed mechanisms, ScalarLens AUC is 0.8383/0.7969/0.7983 versus DAES at 0.8296/0.7463/0.7703, while focal-coordinate displacement stays exactly zero.Categorical logit MSE falls from 0.2891 to 0.1349. On rare Criteo contexts, ScalarLens ranks first in all four DNN/DCNv2 groups, with 0.0012–0.0016 margins versus 0.0007 in common contexts.Source: §4.4 · p. 7, Figure 3 · p. 7

4.5 Parameter Sensitivity

  • Across six dataset–backbone combinations and three seeds, two or three response steps recover T=1’s deficit; other parameters show shallow plateaus, supporting the interior default (16, 3, 16, 32).The apparent gain at K=8 is concentrated in one setting, and K=32 fails the promotion rule; validation results do not revise test rankings.Source: §4.5 · p. 7, Figure 4 · p. 8

4.6 Accuracy and Efficiency

  • Optimization lowers training latency from 1.34× to 0.74× DEER and inference latency from 1.40× to 1.13×, without changing AUC.The remaining 13% inference premium reflects additional computation for contextual responses; optimized execution reduces training latency below DEER’s while approaching, but not surpassing, it at inference.Source: §4.6 · p. 7, Figure 5 · p. 8

5 Discussion and Limitations

The discussion weighs the evidence for the approach against its deployment scope and limits. Its operational argument concerns lifecycle costs, and its evaluation does not establish transfer to unseen families or temporal settings.

5 Discussion and Limitations

  • The approach is not a universally dominant predictor: categorical context is reliable, while numerical context is auxiliary on real data but decisive in the controlled mechanism.AutoInt, ContextFiLM, and FrozenMessage are exceptions where simpler designs remain competitive.Source: §5 · p. 8
  • The original-scale advantage concerns lifecycle costs from transformed copies and online synchronization, not normalization arithmetic itself.The system still requires schema validation and performs interpolation and contextual computation; scale handling is consolidated in the model artifact.Source: §5 · p. 8
  • Because the 27 settings are correlated and development used all datasets, the evaluation does not establish temporal, causal, or unseen-family transfer.Those claims require temporal splits, delayed feedback, and new domains.Source: §5 · p. 8
  • Matched ablations, controlled recovery, frozen rarity groups, and the scale audit jointly support the reported 25/27 result.The evidence hierarchy is stronger than any single statistic; FrozenMessage and the auxiliary numerical path indicate possible deployment simplifications without changing the reported matrix.Source: §5 · p. 8

6 Conclusion

The conclusion frames numerical embedding as separating a value’s stable coordinate from its context-dependent predictive response. In the tested settings, this design ranks first in 25 of 27 dataset-backbone combinations and recovers controlled response mechanisms without moving the coordinate.

6 Conclusion

  • 25 of 27 dataset-backbone combinations rank ScalaRLens first on original numerical scales.The method separates a scalar-determined coordinate from a bounded contextual response while changing only numerical embeddings.Source: §6 · p. 8
  • Ablations rule out scale correction and local capacity as sufficient explanations for the gain.Controlled mechanisms recover categorical, numerical, and mixed responses while the focal coordinate remains invariant.Source: §6 · p. 8
  • The paper concludes that numerical representations should specify an invariant coordinate and an adaptive response rather than make one token serve both roles.The separation is presented as a reusable design principle and an invariant for auditing future CTR encoders.Source: §6 · p. 8

A Controlled Data-Generating Process

The controlled data-generating process defines scalar and categorical inputs, generates labels from a response function, and shifts context prevalence and correlations between training and test while holding that function fixed.

A Controlled Data-Generating Process

  • Each example has three scalar inputs in [-1, 1] and categorical variables with four and three possible values.The scalar vector is x=(x0, x1, x2), with c0 in {0,1,2,3} and c1 in {0,1,2}.Source: A Controlled Data-Generating Process · p. 10
  • Labels are sampled from a Bernoulli distribution whose probability is the sigmoid of q + r s - 0.18.The passage introduces four response mechanisms but does not specify their forms here.Source: A Controlled Data-Generating Process · p. 10
  • The shifted test reverses both category-probability vectors and changes scalar-context correlations while keeping the response function fixed.Training, validation, and IID test use d=1, while shifted test uses d=-1; split sizes are 20k/5k/10k/10k with identical draws across five seeds.Source: A Controlled Data-Generating Process · p. 10

B Evidence for Context Sources

The context-source analysis screens eligible numerical-target/context pairs, tests their residual dependence, and then checks context effects in matched neural interventions and frozen checkpoints.

B Evidence for Context Sources

  • Eligibility requires at least three nonempty target bins and a context with at least three groups; 13/23 AutoML-A, 6/6 AutoML-E, and 11/13 Criteo targets pass.Candidates are selected using training data, refitted on the full training sample, and evaluated on validation data without using test labels.Source: B Evidence for Context Sources · p. 10
  • The joint estimator improves validation log loss for 43.0% of 663 AutoML-A pairs, 73.3% of 150 AutoML-E pairs, and 83.6% of 286 Criteo pairs.Selected categorical and numerical contexts are positive for 27/30 and 29/30 targets; the table compares the joint estimator with an additive model containing both marginal effects.Source: B Evidence for Context Sources · p. 10, Table 7 · p. 10

B.1 Matched neural checks and interventions on frozen checkpoints

  • Categorical context is positive in every paired neural run, while other numerical fields have a smaller effect that depends on the setting.The 2 × 2 validation-AUC study pools AutoML-A/E, DCNv2/DeepFM, and three seeds.Source: §B.1 · p. 10, Table 8 · p. 10
  • Permuting numerical context within focal-value bins changes the reinjected embedding in all settings, but predictive reliance is concentrated on Criteo.The frozen-checkpoint table reports AUC drop or log-loss increase, embedding relative RMS change, and seeds with increased log loss.Source: §B.1 · p. 10, Table 9 · p. 10
  • The rare-context stress-test table reports test AUC mean±std over three seeds and defines ΔF as rare-group AUC relative to Field.Training frequencies define group rarity, with validation covariates setting common and rare thresholds.Source: Table 10 · p. 11

C Detailed Context and Robustness Audits

Categorical and numerical context do not contribute uniformly across paired runs, so observed context dependence is not evidence of universal model utility. The audits also document reproducibility controls and the scope of the claim.

C Detailed Context and Robustness Audits

  • Categorical context is positive in all 18 paired runs, while numerical context is smaller and setting-dependent; predictive reliance is concentrated on Criteo.Permuting context at a frozen checkpoint changes the reinjected numerical token throughout.Source: C Detailed Context and Robustness Audits · p. 11
  • Validation covariates set rarity thresholds at the 20th and 80th percentiles, yielding 916,010 common and 918,284 rare test examples.The thresholds are applied unchanged to the test set.Source: C Detailed Context and Robustness Audits · p. 11

D Reproducibility and Scope

  • Each full matrix contains 1,539 runs, with shared partitions, inputs, optimizer, seeds, early stopping, and test evaluation only after model selection.Aggregation requires records for all three seeds before ranking a method.Source: D Reproducibility and Scope · p. 11
  • Methods replace only the numerical representation interface; categorical embeddings stay unchanged, and vocabularies, ranges, and quantile boundaries are fitted without test labels.Test metrics use checkpoints selected on validation data.Source: D Reproducibility and Scope · p. 11
  • The artifact includes the module, preprocessing entry point, manifests, configurations, per-seed records, scripts, latency traces, and equivalence tests; memory efficiency is unclaimed.The frozen harness did not record peak allocated memory.Source: D Reproducibility and Scope · p. 11
  • The tested matrix supports a reliable interface, not a guarantee that every context source, response mechanism, or consumer benefits.The authors retain non-winning AutoInt settings, mixed numerical-context interventions, and weak FrozenMessage results.Source: D Reproducibility and Scope · p. 11

D.1 Benchmark on original numerical scales and secondary standardization audit

On original numerical scales, ScalarLens ranks first in 25 of 27 dataset–backbone combinations and second in the other two. Its advantage remains significant against three baselines after shared standardization, alongside controlled response-recovery and ablation evidence.

D.1 Benchmark on original numerical scales and secondary standardization audit

  • On original scales, ScalarLens ranks first in 25 of 27 dataset–backbone combinations and second in the remaining two, with mean rank 1.074.It wins eight of nine backbones on AutoML-A, eight of nine on AutoML-E, and all nine on Criteo; mean AUC exceeds the strongest competitor by 0.0070, 0.0037, and 0.0012, respectively.Source: §D.1 · p. 11
  • After shared training-fit z-scoring, ScalarLens wins 18/27 paired cells versus DEER, 21/27 versus DAES, and 25/27 versus NaryDis; all comparisons are Holm-significant.The audit removes all 60 cells below 0.55 AUC from the original-scale evaluation; Table 11 reports the shared z-score results.Source: §D.1 · p. 11, Table 11 · p. 12
  • With a shared linear consumer and unchanged conditional label rule, ScalaRLens achieves shifted-test AUCs of 0.8383/0.7969/0.7983 for categorical/numerical/mixed mechanisms, versus DAES’s 0.8296/0.7463/0.7703.Source: §D.1 · p. 12
  • Across six frozen Criteo rare-group pairs, gains over DAES, DEER, and NaryDis are +0.0015, +0.0014, and +0.0015 AUC; optimized training/inference latency is 0.74×/1.13× DEER.The rare-group margins over the strongest baseline are larger than those for common contexts.Source: §D.1 · p. 12
  • Matched removals lose 0.0038 AUC for Local Only, 0.0037 without normalization, 0.0025 with stopped contextual gradients, and 0.0011 with a one-step response, on average.DEER+Norm and LocalMLP each lose about 0.0043 AUC on average; ContextFiLM loses 0.0011 and wins only one of six settings.Source: §D.1 · p. 12

Related papers

Loading 2609.29182v1…