Source-linked AI summary
SNIP++: Fine-Grained Symbolic-Numerical Alignment for Symbolic Regression
Benjamin Léger, Shubham Gupta, Samy Mammeri, Kazem Meidani, Cem Subakan, Christian Gagné
TL;DR
Global symbolic–numerical alignment does not represent how individual expression parts relate to behavior or how local edits affect it. SNIP++ adds structural positions and multi-granularity behavioral supervision for subexpressions and complete expressions. The resulting representations improve alignment and local discrimination and transfer to external SR corpora.
Problem
Global alignment of complete expressions with their behaviors does not directly supervise individual subtrees or the behavioral consequences of modifying them, leaving fine-grained local functional structure underdetermined.
Source: §1 · p. 1, §3.3 · p. 3Method
SNIP++ combines tree-structural positional encoding with contrastive alignment of contextual and standalone subtree representations to their computed behaviors, plus expanded complete-expression supervision.
Source: §1 · p. 1, §4.2 · p. 4, §4.3 · p. 4Results
SNIP++ improves global alignment and fine-grained discrimination, transfers to held-out edit families and external SR benchmarks, and raises nDCG@10 for f →y from 47.8% to 87.3%.
Source: §1 · p. 2, §5.2 · p. 5, §6 · p. 7Takeaways & Limitations
The representation gains provide a more functionally meaningful shared space for symbolic–numerical correspondence, including local edits and external SR corpora.
Source: §1 · p. 2, §6 · p. 7The paper evaluates the representation itself, leaving its effects on end-to-end generation and search for future work.
Source: §6 · p. 7
Abstract
from arXiv · showhide
Mathematical expressions and the numerical behavior they produce are two views of the same underlying function, and connecting them is central to scientific discovery. Symbolic Regression (SR) relies on this connection directly: it searches for an expression that reproduces a given behavior. Recent multi-modal models learn this connection by embedding symbolic expressions and their numerical behavior in a shared representation space. We show that this embedding space is only globally aligned: complete expressions correspond to complete behaviors, but the contribution of individual parts of an expression is not represented. This granularity gap leaves the model unable to tell how a local edit to an expression changes its behavior, the central operation in SR. We introduce a compositional alignment method that closes this gap: a structural positional encoding exposes the substructure of an expression to the encoder, and a multi-granularity contrastive objective grounds each subexpression in the behavior it produces before propagating this grounding to the full expression. The resulting representations close much of the modality gap between symbolic and numerical embeddings, reliably distinguish the effects of local edits that the original alignment cannot, and transfer to external SR corpora.
1 Introduction
Symbolic regression connects expression form with observed numerical behavior, but global alignment does not capture how expression parts relate to behavior or how local edits change it. SNIP++ adds compositional grounding and improves alignment and local discrimination, including on held-out edit families and external SR benchmarks.
1 Introduction
- Small expression changes can leave behavior nearly unchanged or alter it completely, and distinct symbolic forms can produce the same behavior on finite observations.This makes learning the inverse and relational structure of the correspondence central to symbolic regression.Source: §1 · p. 1
- SNIP’s pooled global contrastive alignment captures coarse cross-modal compatibility but cannot reliably distinguish closely related symbolic variants.The paper calls this failure to resolve fine-grained structure–behavior relations the granularity gap.Source: §1 · p. 1
- SNIP++ combines tree-structural positional encoding with multi-granularity supervision of subexpressions and controlled symbolic perturbations.The method grounds parts in their numerical behavior and teaches which structural edits alter behavior.Source: §1 · pp. 1–2
- SNIP++ substantially improves global symbolic–numerical alignment and fine-grained local discrimination, reduces the modality gap, and transfers to held-out edit families and external SR benchmarks.The representation space is reported to become more functionally meaningful.Source: §1 · p. 2
2 Related Work
Prior work has explored direct expression generation, shared symbolic–numerical representations, fine-grained multimodal alignment, and structure-aware symbolic encoders. SNIP++ complements these approaches by improving cross-modal alignment of local symbolic changes and components.
Neural and multimodal symbolic regression
- SNIP++ focuses on improving alignment itself, complementing approaches that use decoders or search procedures for expression generation and latent-space search.The improved alignment is intended to support candidate comparison, evaluation, and search guidance.Source: §2 · p. 2
Fine-grained and structure-aware representations
- SNIP++ combines tree-structural positions with computational subtrees as local units grounded in their contextual and standalone numerical behaviors.This joins structure-aware Transformers and fine-grained cross-modal alignment approaches.Source: §2 · p. 2
Locality and compositional search
- Unlike methods that prescribe compositional search operators or heuristics, SNIP++ learns a bidirectional symbolic–numerical representation of components as a reusable semantic signal.Prior work includes genetic-programming approaches to semantic locality and RAG-SR’s component-retrieval perspective.Source: §2 · p. 2
3 Background and Problem Formulation
The paper frames symbolic–numerical alignment as a requirement at both complete-expression and component levels. SNIP supplies a global-alignment baseline, while the granularity gap describes what that global objective leaves unconstrained.
3.1 SNIP
- SNIP aligns pooled complete-expression and numerical-behavior embeddings bidirectionally with an InfoNCE objective.Its dual encoders support symbolic-to-behavior and behavior-to-symbolic retrieval.Source: §3.1 · p. 2
- SNIP serializes expression trees in prefix order and pools token representations, providing a controlled backbone for studying alignment independently of a decoder or search procedure.The two retrieval directions support candidate comparison and expression recovery.Source: §3.1 · p. 2
3.2 Alignment Requirements
- A representation useful for symbolic regression should capture functional correspondence both for complete expressions and for their constituent components.Complete-expression alignment requires matching an expression with its generated behavior while distinguishing behaviors from other functions.Source: §3.2 · p. 2
- Fine-grained alignment requires grounding each subtree in its computed intermediate behavior while preserving its boundaries and position within the expression.It must also represent context-dependent contributions because operations above a subtree transform its output.Source: §3.2 · p. 3
- Fine-grained alignment should separate nearby expressions when edits change behavior while keeping syntactically different expressions close when they represent the same function.The representation must capture both functional sensitivity and context-dependent contribution.Source: §3.2 · p. 3
3.3 Granularity Gap
- SNIP’s global objective constrains pooled expressions, not individual subtrees or edit consequences, and sequential positions do not expose tree relations or subtree membership.The paper describes this underdetermination of local functional structure as the granularity gap.Source: §3.3 · p. 3
4 Proposed Method
SNIP++ adds explicit tree structure and local behavioral supervision to SNIP’s symbolic–numerical alignment. It grounds subtree representations in their computed behaviors, then expands global supervision so local edits also affect complete-expression representations.
4 Proposed Method
- SNIP++ modifies SNIP with tree-structural positional encoding and multi-granularity contrastive supervision while leaving the numerical encoder, tokenizer, and pooling operation unchanged.Training samples expressions, subtrees, and perturbations and jointly optimizes the objective in one stage; inference uses frozen encoders.Source: §4 · p. 3, §4.3 · p. 4
4.1 Tree-Structural Positional Encoding
- Tree-structural positional encoding augments each token with its location in the expression tree, making parent–child relations and subtree membership available to the symbolic encoder.It records nearest branch choices toward the root; the nearest relation is indexed k = 0, with learned channel weights for farther relations.Source: §4.1 · pp. 3–4, Figure 1 · p. 3
4.2 Fine-Grained Subtree Alignment
- The local objective aligns each subtree’s numerical behavior with both its contextual representation within the parent and its standalone symbolic representation.Original and perturbed subtrees are treated symmetrically, grounding each subtree in what it computes and how it appears within its parent.Source: §4.2 · p. 4
4.3 Complete-Expression Alignment
- The expanded global objective adds perturbed parent-expression pairs so each local edit supervises how its numerical effect appears in the pooled complete expression.This complements local subtree supervision, which does not by itself require the parent representation to capture the edit’s effect.Source: §4.3 · p. 4
5 Experimental Evaluation
The evaluation moves from complete-expression retrieval to local edit discrimination, external-corpus transfer, functional probes, and component analysis. It assesses alignment directly through frozen representations.
5 Experimental Evaluation
- The evaluation isolates alignment using frozen encoders and cosine similarity, progressing from global retrieval to fine-grained discrimination, transfer, functional probes, and cumulative ablation.Global retrieval tests 100,000 held-out expression–behavior pairs.Source: §5 · p. 4
5.1 Experimental Setup
- Evaluation uses independently sampled expressions and compares SNIP++ with a SNIP baseline trained under identical conditions.Global alignment uses Recall@10 and nDCG@10 in both directions; modality gap is the distance between normalized embedding centroids.Source: §5.1 · p. 5
5.2 Global Symbolic–Numerical Alignment
- nDCG@10 rises from 47.8% to 87.3% for f →y and from 31.3% to 82.6% for y →f, improving global alignment in both directions.The nDCG@10 directional gap falls from 16.5 to 4.7 points.Source: §5.2 · p. 5, Table 1 · p. 5
5.3 Fine-Grained Symbolic Discrimination
- In local fine-grained symbolic discrimination, SNIP++ averages percentile ranks of .73 for f →y and .79 for y →f across edit types, versus SNIP’s .48 and .69, respectively.The gains persist across three held-out transformation types; SNIP++ has a higher point estimate in 11 of 12 edit-type–direction settings, with the sole reversal for y →f under full-subtree rewriting.Source: §5.3 · pp. 5–6
5.4 Out-of-Domain Generalization
- Across Feynman, Classic, Strogatz, and their pooled union, SNIP++ averages percentile ranks of .79 for global alignment and .69 for local discrimination, versus SNIP’s .47 and .48.SNIP++ remains above chance across all corpora and directions in both protocols.Source: §5.4 · p. 6, Figure 4 · p. 6
5.5 Functional Structure in the Learned Space
- SNIP++ shifts the learned geometry toward function: it separates behavior-changing edits, ranks equivalent rewrites at .90 versus .72, and raises Taylor-approximation similarity from .24 to .45.For continuous variation, similarity between sin(5x) and sin(0.5x) falls from .94 with SNIP to .06 with SNIP++.Source: Figure 5 · p. 6, §5.5 · p. 7
5.6 Cumulative Component Analysis
- Contextual subtree alignment produces the largest fine-grained discrimination gains; standalone subtree alignment and symmetric perturbation supervision improve directional balance.Tree positional encoding alone changes little, while complete-expression propagation carries gains to expression-level representations.Source: §5.6 · p. 7, Figure 6 · p. 7
6 Conclusion and Future Work
SNIP++ combines tree-structural positional encoding with multi-granularity behavioral supervision, improving global alignment and fine-grained discrimination while transferring to external SR corpora. The paper evaluates the representation itself, leaving its effects on end-to-end generation and search for future work.
6 Conclusion and Future Work
- SNIP++ combines tree-structural positional encoding with multi-granularity behavioral supervision to align symbolic expressions and numerical behavior.The paper reports gains in global alignment and fine-grained discrimination, a narrower modality gap, and transfer to external SR corpora.Source: §6 · p. 7
- Ablations show that tree structure alone is insufficient; the largest improvements arise when subtrees are aligned with the behaviors they compute.Encoding subtrees both within their parent and independently is especially helpful for numerical-to-symbolic alignment, where multiple symbolic forms can express the same behavior.Source: §6 · p. 7
- The study evaluates the representation rather than embedding SNIP++ in a complete symbolic-regression system, leaving end-to-end effects untested.Future work proposed includes using subtree representations to guide search, adapting encoders for generative SR, and developing a reference-free comparison score.Source: §6 · p. 7
A PCA Geometry and Variance Spectrum
SNIP++ distributes variance more evenly across leading dimensions and reduces symbolic–numerical centroid separation. The supplied methods also detail tree-structural encoding and contrastive objectives at subtree and complete-expression levels.
A PCA Geometry and Variance Spectrum
- 12.4% of variance lies in SNIP++’s first two principal components, versus 41.6% for SNIP; centroid distance falls from 1.18 to 0.31.For SNIP++, the first two components explain 6.5% and 5.9%; for SNIP, they explain 35.0% and 6.6%.Source: A PCA Geometry and Variance Spectrum · p. 10
- SNIP++ reaches 50% cumulative variance in 11 dimensions versus four for SNIP, but needs 48 rather than 91 dimensions to reach 99%.The models need similar dimensionality for 90% and 95% cumulative variance: 30 and 37 dimensions for SNIP++, versus 26 and 35 for SNIP.Source: A PCA Geometry and Variance Spectrum · p. 10
B.1 Overview
- Tree-structural positional encoding augments each symbolic token with a vector derived from its expression-tree node, alongside SNIP’s sequential position encoding.The vector represents nearest parent–child relations on the node-to-root path, using child slots and zero padding for missing relations.Source: §B.1 · p. 10, §B.2 · p. 10
B.2 Constructing the Structural Vector
- The structural vector uses multiple learned decay channels, retaining nearer branch relations while allowing channels to represent different ancestral ranges.The nearest relation has weight one; farther relations receive progressively decayed weights, and the channel vectors are concatenated.Source: §B.2 · pp. 10–11
B.3 Token Assignment
- Every tokenizer piece for a tree node receives that node’s structural vector; special and padding tokens receive zero vectors.The encoding is applied only to the symbolic encoder; the numerical encoder and the remaining Transformer and pooling components are unchanged from SNIP.Source: §B.3 · p. 11
C.1 Training Examples and Representations
- Training pairs each original or perturbed subtree’s contextual and standalone symbolic representations with the numerical behavior it computes.Variants replace the selected subtree at its original location, and parent expressions and variants are evaluated on the same inputs.Source: §C.1 · pp. 11–12
C.3 Complete-Expression and Subtree Objectives
- The local objective aligns numerical subtree representations with both contextual and standalone symbolic representations, alongside complete-expression alignment.The full objective combines the global complete-expression loss and local subtree losses.Source: §C.3 · p. 12
C.4 Cumulative Objectives
- The cumulative objectives add contextual subtree alignment, standalone subtree alignment, symmetric supervision for perturbed subtrees, and perturbed complete-expression alignment.At the symmetric stage, each retained perturbation is a positive example in both retrieval directions; the final stage also aligns the corresponding complete expressions.Source: §C.4 · pp. 12–13
D Controlled-Edit Evaluation Details
The controlled-edit evaluation compares original expressions with edited variants on shared inputs, retaining only valid, finite variants with distinct numerical behavior. It scores how well each alignment direction ranks the correct pairing against edited alternatives.
D.1 Targets and Input Samples
- The evaluation samples 128 held-out targets, assigns each 200 inputs, and evaluates originals and variants on the same inputs.After requiring finite outputs and variance of at least 10^-10, 127 complete-expression targets remain.Source: §D.1 · p. 13
D.2 Controlled Edit Families
- Six edit families include operand reordering, full-subtree rewriting, unary and binary operator swaps, variable substitution, and tree reshaping.Three families overlap with training edits and three are held out; target expressions and individual edit instances remain unseen during training.Source: §D.2 · pp. 13–14
D.3 Variant Generation and Filtering
- For each target and applicable family, up to 64 candidates are generated and filtered for valid trees, finite outputs, and behavior distinct from the original and retained variants.Output vectors are compared exactly after conversion to float32 on the shared 200 inputs, so retained variant counts can differ by target and family.Source: §D.3 · p. 14
D.4 Evaluation-Pool Statistics
- A target contributes to a family’s evaluation pool only when at least one valid variant is retained.The supplied Table 4 caption describes target counts and mean and median retained variants, but provides no cell values.Source: §D.4 · p. 14, Table 4 · p. 14
D.5 Percentile Rank and Aggregation
- Percentile rank measures the fraction of edited alternatives that score below the correct pairing; 1 is perfect and .5 is expected under random ranking.In f →y, the original expression ranks its behavior against edited behaviors; in y →f, the original behavior ranks its expression against edited expressions.Source: §D.5 · p. 14
- Scores are averaged across contributing targets within each family and direction, then family-level scores are averaged for training-represented or held-out families.Confidence intervals use 1,000 bootstrap resamples of target expressions.Source: §D.5 · p. 14
E Contextual-Subtree Discrimination
SNIP++ improves contextual-subtree discrimination over SNIP across all six edit families, including families held out during training. The evaluation compares original and edited subtrees at the same location within the same parent structure.
E Contextual-Subtree Discrimination
- The evaluation selects up to eight non-trivial subtrees from each of 128 held-out parent expressions and generates retained variants using six edit families.Each edited subtree is inserted at the original tree position, and its contextual symbolic representation is paired with its numerical behavior on the same inputs.Source: E Contextual-Subtree Discrimination · pp. 14–15
- In each direction, the original contextual representation or behavior ranks its correct match against the edited alternatives, with scores averaged first within parents and then across parents.Bootstrap resampling and paired model comparisons keep all subtrees and variants from a parent together.Source: E Contextual-Subtree Discrimination · p. 15
- .87 and .92 are SNIP++’s average percentile ranks for f →y and y →f, versus .47 and .61 for SNIP.The gains occur across all six edit families, including held-out tree reshaping, operand reordering, and full-subtree rewriting; SNIP++ is significantly better in all 12 family-direction settings.Source: E Contextual-Subtree Discrimination · p. 15
F Out-of-Domain Evaluation Details
On three external symbolic-regression corpora and their pooled union, SNIP++ transfers its local alignment more effectively than SNIP. The evaluation also tests controlled edits in both retrieval directions and reports results by edit family.
Evaluation protocol
- Evaluation covers Feynman, Classic, and Strogatz corpora, plus their pooled union, using pooled retrieval and controlled-variant discrimination in both f →y and y→f directions.Inputs are standardized before numerical encoding, with the same standardized inputs used for SNIP and SNIP++.Source: F Out-of-Domain Evaluation Details · p. 15
Scoring and uncertainty
- Scores are averaged within each parent expression and then across parents, with confidence intervals from 1,000 bootstrap resamples of parent expressions.For contextual-subtree evaluation, the parent encoding’s subtree span is matched to the numerical behavior produced by that subtree.Source: F Out-of-Domain Evaluation Details · p. 15
Transfer results
- SNIP++ averages .68 versus SNIP’s .47 in fine-grained discrimination, and .78 versus .48 in pooled contextual-subtree retrieval.These results closely track complete-expression scores of .69 and .79, respectively.Source: F Out-of-Domain Evaluation Details · p. 15
Edit-family results
- Transfer is strongest for variable substitutions and operator swaps, with improvements also extending to structural edit families held out during training.Figure 8 reports correct-pairing percentile rank across six edit families; shaded rows average the three represented and three held-out families, with 95% bootstrap intervals over parent expressions.Source: F Out-of-Domain Evaluation Details · p. 15, Figure 8 · p. 15
G Training-Time Perturbation Construction
Training-time perturbations create nearby symbolic alternatives by changing one controlled choice within a sampled subtree. Valid variants provide both local behavior contrasts and additional symbolic–numerical training pairs.
Evaluation scope
- Evaluation uses the three training edit families plus three held-out families, with all evaluation expressions and edit instances independently generated and unseen during training.Table 5 reports out-of-domain pool counts after validity filtering.Source: G Training-Time Perturbation… · p. 15, Table 5 · p. 16
Perturbation families
- Each training expression supplies one sampled non-trivial subtree, which is perturbed using unary-operator swaps, variable substitutions, or binary-operator swaps.Unary swaps use compatible operator groups; substitutions replace all occurrences of one variable with another already present, and binary swaps change one operator while retaining its operands and remaining subtree.Source: G Training-Time Perturbation… · p. 16
Sampling and validity
- Each edit changes one symbolic choice while leaving the rest of the subtree structure unchanged.Two attempts are made per subtree; inapplicable edits, malformed expressions, and variants that cannot be evaluated are discarded.Source: G Training-Time Perturbation… · p. 16
Behavioral filtering
- The original subtree and up to two retained variants are evaluated on the same input set, and variants must have finite outputs distinct from the original and previously retained behaviors.Exact behavioral duplicates are removed by hashing output vectors.Source: G Training-Time Perturbation… · p. 16
Training-objective role
- Retained perturbations serve as hard numerical alternatives, are encoded symbolically within their parent and independently, and are paired with their own numerical behavior.Inserting a perturbation into its parent also supplies an additional complete-expression training pair.Source: G Training-Time Perturbation… · p. 16
H Implementation Details
SNIP++ keeps the SNIP backbone and adds four learned tree-decay scalars for structural positional encoding. The implementation uses online-generated training expressions, a selected subtree-loss configuration, and frozen encoders for evaluation.
Data and training
- Training expressions are generated online with maximum input dimensionality 10, and all experiments train the models from scratch with seed 0.SNIP++ trains for 100 epochs with 1,000 optimizer updates per epoch, using a global batch size of 64 across four H100-80GB GPUs.Source: H Implementation Details · p. 16
Architecture
- SNIP++ retains SNIP’s numerical encoder, tokenizer, pooling operations, and projection heads; it adds only four learned tree-decay scalars, with approximately 67.2M parameters for SNIP and 67.2M plus four scalars for SNIP++.All experiments use the SNIP backbone; both symbolic and numerical encoders have eight Transformer layers, dimension 512, feed-forward dimension 2048, and 16 attention heads.Source: H Implementation Details · p. 16
Hyperparameter selection
- The selected configuration uses λsub = 1.0 and τs = 0.1, chosen on held-out fine-grained alignment without regression in global retrieval.The search evaluated λsub ∈{0, 0.5, 1.0} and τs ∈{1.0, 0.1}.Source: H Implementation Details · p. 16
Compute
- A complete SNIP++ run takes approximately 21.4 hours on four H100-80GB GPUs, or roughly 86 GPU-hours, while SNIP takes 11.4 hours on one such GPU.Evaluation uses one H100-80GB GPU, eight CPU cores, and 64GB of system memory; all evaluations use frozen encoders.Source: H Implementation Details · p. 16