PaperlayerView exact revision on arXiv

Source-linked AI summary

Universal Sample-Based Embeddings of Higher-Order Objects

Radovan Haluška, Christopher Solinas, Nathan R. Sturtevant, Martin Schmid, Michael Bowling

arXiv:2610.19899v1cs.LGcs.AI

TL;DR

Higher-order objects lack compact vector representations, and there is no general established solution for learning across families, including unseen members. The paper introduces a sample-based encoder with object-specific decoders and evaluates it on distributions, functions, and graphs. Across these settings, the supplied results describe family-level reconstruction and graph edge separation, alongside function models that capture varied behaviors and scale to 32 dimensions.

  • Problem

    Higher-order objects lack natural compact vector representations, and learning across their families when parameterizations are unknown has no general established solution.

    Source: §1 · p. 1

  • Method

    A sample-based encoder maps each object to a fixed-size embedding that conditions a domain-specific fitting decoder, trained using an object-specific loss.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    A single model captures the reported function families up to 32 dimensions, while graph embeddings separate edges from non-edges across four settings and distribution embeddings outperform KDE on five metrics.

    Source: §3 · p. 5, §4 · p. 6, §5 · p. 8

  • Takeaways & Limitations

    The embeddings can make ordinary neural networks act as neural operators over functions and support reconstruction and prediction across families of higher-order objects.

    Source: §5 · p. 8, §6 · p. 9

    The framework requires an object sampler that supplies samples and a parameterizable fitting model capable of representing an object from samples.

    Source: §2 · p. 2

Abstract

from arXiv · show

Neural networks operate naturally on finite-dimensional vectors, yet many problems require reasoning about higher-order objects such as distributions, functions, and graphs. Such objects capture relationships within a ground set and rarely admit an explicit compact representation. Moreover, we typically want a compact representation not of a single higher-order object but of an entire family of such objects, including members unseen during training. While a few approaches exist for particular object types, none is designed to work across object types. We propose a unified framework that represents any higher-order object as a fixed-size embedding vector learned over a family of such objects. An encoder maps an object, seen only through a finite set of samples, to an embedding, and a domain-specific decoder supplies the training signal by modeling the object conditioned on that embedding. Since the encoder works only from samples, one architecture serves different object types. Because the decoder is a model already suited to single instances of the object type, the framework inherits that model's native operations and inductive biases. We instantiate the framework on distributions, functions, and graphs. In each case we show that a single trained model represents an entire family and generalizes to members unseen during training. Furthermore, we demonstrate the downstream utility of these embeddings: interpolating distributions in embedding space, solving differential equations in embedding space, and predicting unseen edges from a graph's embedding.

1 INTRODUCTION

Higher-order objects such as distributions, functions, and graphs lack natural compact vector representations, and learning across their families—including unseen members—has no general established solution. The paper proposes a sample-based framework with a shared encoder and object-specific decoders, instantiated across these object types.

1 INTRODUCTION

  • Higher-order objects such as distributions, functions, and graphs lack natural compact representations for neural networks, which otherwise operate on finite-dimensional vectors.This makes it unclear how to use an input distribution, for example, to predict its probability density function.Source: §1 · p. 1
  • Learning and manipulating families of higher-order objects, especially when their parameterization is unknown, has no general established solution.Existing approaches are typically ad hoc and domain-specific.Source: §1 · p. 1
  • Earlier approaches target individual object types and use inductive biases that do not transfer across types.The cited examples include distribution embeddings, function-space models, and graph methods that often embed nodes or generate graphs rather than representing individual graphs.Source: §1 · p. 2
  • The framework maps finite samples of an object to a fixed-size embedding and uses a domain-specific decoder to provide the training signal.A sample-based encoder can serve different object types, while the decoder brings its native operations to the family and yields reusable representations.Source: §1 · p. 2
  • The framework is instantiated on distributions, functions, and graphs, with the decoder changing primarily across object types.Reported uses include distribution arithmetic and interpolation, ODE solving in function-embedding space, and predicting graph edges from sampled edges.Source: §1 · p. 2

2 EMBEDDING & LEARNING FROM FAMILIES OF HIGHER-ORDER OBJECTS

The framework learns fixed-size representations across a family by encoding sample sets and training a fitting decoder on held-out samples. It requires an object sampler, a model capable of fitting an object, and a training loss; its decoder also brings object-specific operations to family members.

2 EMBEDDING & LEARNING FROM FAMILIES OF HIGHER-ORDER OBJECTS

  • The framework requires an object sampler, a parameterizable fitting model, and an object-dependent training loss.The sampler supplies the only access to an object through samples; the fitting model learns to represent it from those samples.Source: §2 · p. 2
  • The embedder compresses samples into a fixed-size vector that can condition a decoder or serve as input to and output from other networks.Sample-only inputs allow one architecture to handle different object types and sample-set sizes, including objects unseen during training.Source: §2 · p. 3
  • Using a decoder suited to fitting a single object gives the framework that decoder’s training signal and native operations across a family.Examples include sampling and density estimation in Flow Matching models.Source: §2 · p. 3
  • A family is treated as a distribution over sample sets, and one model learns to fit that distribution and generalize across its members.Each set describes an object; multiple overlapping sets of varying size may represent the same object, without assumptions such as a regular grid.Source: §2 · p. 3
  • Each sample set is split so the embedder encodes one half and the decoder is scored on the held-out half, with both networks trained jointly.The split discourages memorization and encourages an embedding that captures transferable structure.Source: Figure 1 · p. 3, §2 · p. 4
  • The framework is evaluated on families of distributions, functions, and graphs, provided the three required components are available.For a donut family, the object sampler supplies points and the fitting model can be a generative model trained with negative log-likelihood.Source: §2 · pp. 3–4

3 DISTRIBUTION EMBEDDINGS

For distributions, a sample-set embedder conditions a generative decoder, trained by scoring held-out samples. On donut distributions, the learned representation reconstructs densities and samples, compares favorably with KDE, and reflects the family’s low-dimensional structure.

3 DISTRIBUTION EMBEDDINGS

  • With either decoder, the approach outperforms KDE on all five metrics and trails the parametric fit only marginally on radius and width recovery.The parametric fit is given the donut family’s form, whereas the proposed approach works from samples without that knowledge.Source: Table 1 · p. 5, §3 · p. 5
  • From 32 samples, the embedding lets the decoder reproduce a donut distribution as both fresh samples and a density.The evaluation uses a Set Transformer embedder; its embedding conditions a generative decoder trained on held-out samples.Source: §3 · pp. 4–5
  • Although embeddings are 16-dimensional, the first four principal components explain 99.5% of flow-matching embedding variance and 97.7% for the normalizing flow.Each component aligns almost one-to-one with a generative property, with |r| ≥ 0.95; the paper also explores embedding arithmetic and interpolation.Source: §3 · p. 5

4 FUNCTION EMBEDDINGS

Function embeddings encode sample sets into fixed-size vectors that parameterize pointwise decoders, enabling reconstruction and operator-based prediction of functions. Across three function families, a single model captures varied behavior and scales to 32-dimensional states.

4 FUNCTION EMBEDDINGS

  • A function is represented by input–output samples mapped to a fixed-size embedding, which conditions a pointwise decoder trained on held-out samples.The embedder and decoder are jointly trained to minimize prediction error on the held-out half of each sample set.Source: §4 · p. 5
  • Embedding-space operators map polynomial embeddings to derivative embeddings and map ODE parameters and initial conditions to solution embeddings.Ordinary feedforward networks perform these mappings, and the decoder expands their outputs into functions.Source: §4 · p. 6
  • From small Lotka–Volterra sample sets, the decoder reconstructs trajectories, while an operator predicts solutions from initial conditions and parameters alone.The same decoder recovers trajectories from either sample-inferred or operator-predicted embeddings.Source: §4 · p. 6, Figure 3 · p. 6
  • A single model captures the varied behaviors of polynomial, damped-oscillator, and Lotka–Volterra families, scaling to functions with state dimensions up to 32.The oscillator and Lotka–Volterra solutions are oscillatory, while polynomials span constants through cubics.Source: §4 · p. 6
  • The ODE solution operator can outperform noisy sample-inferred embeddings, whereas the differential operator inherits sample noise and adds its own.The differential operator's error exceeds that of the embedding it takes as input.Source: §4 · pp. 6–7

5 GRAPH EMBEDDINGS

Graphs are represented by sampled edges encoded into a fixed-size embedding that conditions a decoder of vertex Laplacian representations. The learned geometry separates edges from non-edges, while threshold-based reconstruction degrades on larger graphs and is harder for scale-free networks.

5 GRAPH EMBEDDINGS

  • A graph is encoded from randomly sampled edges into a fixed-size embedding that conditions a decoder of vertex Laplacian representations.The sampled edges are split between embedding formation and decoder evaluation; the networks are trained jointly.Source: §5 · p. 7
  • The decoder conditions both vertex representations and the dual-variable network on the embedding, allowing one model to produce Laplacian representations across a graph family.This replaces fitting the representation network and dual-variable matrix separately to each graph.Source: §5 · p. 7
  • The medians network predicts per-vertex median neighbor distances from sampled edges, and reconstruction thresholds pairwise distances using the two endpoints’ medians.The degree-based threshold strategy is an oracle using each vertex’s true degree.Source: §5 · p. 8
  • For sampled trees, the predicted geometry already reveals tree structure before distances are thresholded into a graph reconstruction.Figure 4 encodes smaller representation distances with thicker, darker lines.Source: §5 · p. 8, Figure 4 · p. 8
  • The learned geometry separates edges from non-edges across four tested settings, and the medians network nearly matches the degree oracle on n = 10 families.Both reconstruction strategies degrade for n = 20, largely alongside less accurate geometry; scale-free graphs are harder than trees at both sizes.Source: §5 · p. 8
  • A single median struggles to separate a hub’s edges from non-edges because its neighbors span a wide range of distances.Scale-free graphs are described as denser and containing hubs, making them harder than trees at both tested sizes.Source: §5 · p. 8

6 DISCUSSION

The framework applies one family-embedding approach to distributions, functions, and graphs, whose prior methods have largely been object-specific. The discussion highlights fixed-size representations for downstream use while noting that graph geometry is easier to recover than exact adjacency.

Discussion

  • The framework is demonstrated across distributions, functions, and graphs, object types historically approached with distinct domain-specific methods.Its breadth is presented as generality across otherwise distinct higher-order objects.Source: §6 · p. 8

Distributions

  • Distribution embeddings provide fixed-size belief states for policies and value functions, and extend beyond the continuous distributions previously used by Generative Distribution Embeddings.The framework also generalizes the Neural Statistician by allowing decoders beyond a variational autoencoder.Source: §6 · p. 9

Functions

  • Function embeddings let ordinary neural networks act as operators without requiring fixed grids, sensor locations, or resolutions.A single model maps parameters to ODE solutions, while PDE solving is framed as mapping one function embedding to another.Source: §6 · p. 9

Graphs

  • The graph-family model lifts ALLO from one graph to many by predicting its dual variables from an embedding, and recovers geometry that separates edges from non-edges.Exact adjacency is harder because ALLO optimizes spectral geometry, while per-vertex median thresholds can fail especially at high-degree hubs.Source: §6 · p. 9

7 CONCLUSION

The conclusion presents a framework for learning compact representations across families of higher-order objects, using samples and a fitting model. The supplied sections also outline the generative-model and graph objectives, related approaches, the multi-stage encoder, and the donut family used in experiments.

7 CONCLUSION

  • A single trained model represents a whole family of higher-order objects and generalizes to members unseen during training.The framework uses an object sampler, a parameterizable fitting model, and a training loss; it is instantiated on distributions, functions, and graphs.Source: §7 · pp. 9–10

A.1 GENERATIVE MODELING

  • Normalizing flows transform a tractable base distribution through an invertible differentiable map, while flow matching learns a velocity field and incurs an ODE solve for sampling and density.For flow matching, training regresses toward the target velocity along an optimal-transport interpolant; discrete flow matching operates directly on discrete support without ordinal bias.Source: §A.1 · p. 13

A.2 AUGMENTED LAGRANGIAN LAPLACIAN OBJECTIVE

  • ALLO learns a single graph’s Laplacian eigenvector representation from sampled edges using a saddle-point objective and dual variables.Diagonal dual entries track eigenvalue-proportional values, while off-diagonal entries enforce eigenvector orthogonality.Source: §A.2 · p. 13

B RELATED WORK

  • Prior approaches target particular object types, whereas the framework can recast methods as special cases and use an existing single-object model as decoder.The supplied related-work passages describe distribution, function, and graph approaches as object-specific.Source: B RELATED WORK · pp. 13–14

C MULTI-STAGE SET ENCODER

  • The multi-stage set encoder re-reads original samples with pooled context appended, allowing later stages to compute features relative to set-level context.The wrapper applies to set encoders such as DeepSets and Set Transformer, with weighted-mean or attention pooling.Source: C MULTI-STAGE SET ENCODER · p. 14

D.1 DISTRIBUTION FAMILIES

  • The donut family varies center, radius, and width, and its samples are drawn from distributions over these parameters.The family spans thin and thick rings; its sample set consists of i.i.d. points from each donut.Source: §D.1 · pp. 14–15

D.2 EMBEDDING PROPERTIES

The experiments test whether distribution embeddings support interpolation and vector arithmetic. Interpolated points decode to intermediate donuts, while arithmetic predicts combined parameter changes except where the donut family contains a multiplicative interaction.

Interpolation

  • Interpolated embeddings decode to well-formed intermediate donuts with intermediate center, radius, and width, although those interior embeddings were never produced by the encoder during training.The rings translate, grow, and thin along the embedding segment rather than cross-fading or smearing.Source: §D.2 · p. 15

Embedding arithmetic

  • For independent parameter changes, vector arithmetic predicts the combined donut with relative errors of 3.1% and 4.2%, near the 3.9% sampling-noise floor.The two positional/shape tests have cosine similarity above 0.999; combining radius and width instead misses by 24% because absolute thickness is their product.Source: §D.2 · p. 15

D.3 ADDITIONAL EXPERIMENTS

The additional experiments examine how sample count affects donut reconstruction and describe the two generator heads and evaluation protocol. Reconstruction improves with more samples before saturating, while the heads tie on NLL and differ slightly on SWD.

D.3 ADDITIONAL EXPERIMENTS

  • Reconstruction improves steeply from n = 16 to n = 64 and saturates by n = 128; the two heads tie on NLL, while the normalizing flow is slightly better on SWD at every n.Center and radius errors level off by n = 128, while width error stops improving after n = 32; the sweep uses six sample counts on 1,024 held-out donuts.Source: §D.3 · p. 15, Table 5 · p. 17

D.4 HYPERPARAMETERS

  • The flow-matching head learns a velocity field conditioned on the embedding and samples by integrating its ODE, while the normalizing-flow head uses masked coupling layers and evaluates density in one pass.Both heads receive the embedding by concatenation; the flow head uses a midpoint integration rule.Source: §D.4 · p. 17
  • Evaluation embeds each held-out donut from n samples, scores NLL on fresh samples, and compares generated samples by sliced Wasserstein distance and parameter recovery error.KDE and parametric-fit baselines are fit to the same n samples.Source: §D.4 · p. 17

E.1 FAMILIES

The section defines polynomial and ODE function families spanning varied output scales and state dimensions, then describes shared and family-specific embedding architectures. The dot-product encoder is the default because it performed better than concatenation in the reported comparisons.

E.1 FAMILIES

  • The differential operator is trained only on degree-1–3 polynomials because degree-0 derivatives are zero and undefined under relative-L2 loss.The embedding itself is trained over degrees 0–3.Source: §E.1 · p. 18
  • Damped oscillator trajectories span state dimensions from 2 to 32 as the number of masses varies from 1 to 16.The family has a closed-form solution, z(t) = exp(At) z0.Source: §E.1 · p. 18
  • Lotka–Volterra trajectories are numerically integrated and form periodic orbits with varying amplitudes across initial states and interaction rates.The system has a two-dimensional state and no elementary closed-form solution.Source: §E.1 · p. 19

E.2 ARCHITECTURES & HYPERPARAMETERS

  • The dot-product encoder is the default because its multiplicative modulation of sample features yielded a stronger encoder than concatenation.Both variants use ISAB-based stages and share hyperparameters with the decoder and operators.Source: §E.2 · p. 19
  • The function decoder reconstructs values at query points by attending from an input-derived query over synthetic key and value vectors generated from the embedding.This embedding-as-slots design outperforms directly concatenating the query and embedding into an MLP.Source: §E.2 · p. 19
  • The differential operator maps an embedding to its derivative embedding, while the ODE operator maps initial conditions and parameters to a solution embedding.Both are feedforward MLPs, and the decoder expands predicted embeddings into functions.Source: §E.2 · p. 19

E.3 ADDITIONAL RESULTS

Additional results report lower aggregate reconstruction error for the dot-product encoder and illustrate function reconstruction and differentiation. The graph experiments cover random trees, scale-free networks, and weighted gridworlds, using sampled edges, an embedding-conditioned spectral decoder, and reconstruction heads.

E.3 ADDITIONAL RESULTS

  • Aggregated across polynomial families, the dot-product encoder attains lower reconstruction error than concatenation, with error growing as polynomial degree increases.Figure 9 illustrates reconstruction from samples and differentiation through an operator on embeddings.Source: §E.3 · p. 19, Figure 9 · p. 21

F.1 FAMILIES

  • Graph families include uniformly sampled labeled trees, Barabási–Albert networks with degree-based hubs, and weighted connected subgraphs of a 4 × 4 grid.The gridworld family tests weighted rather than binary adjacency reconstruction.Source: §F.1 · pp. 20–21

F.2 ARCHITECTURES & HYPERPARAMETERS

  • The graph encoder embeds sampled edges, and each edge set is split between producing the embedding and scoring the decoder; configurations use three seeds.Across graph families, only representation dimensions and sampled-edge counts change.Source: §F.2 · p. 21
  • The spectral decoder conditions its vertex representations and dual-variable predictions on the embedding, then drops the constant eigenvector and rescales coordinates by inverse square-root eigenvalues.The decoder is optimized through an augmented-Lagrangian saddle point.Source: §F.2 · p. 22
  • Graph reconstruction uses a medians head or degree oracle for unweighted graphs, and a transition head that predicts normalized adjacency for weighted gridworlds.The transition head is trained on held-out sampled edges; frequency-based sampling identifies weights only up to overall scale.Source: §F.2 · p. 22
  • Evaluation ranks vertex pairs by representation distance for ROC-AUC and average precision, and scores adjacency reconstruction with precision, recall, and F1.Gridworld evaluation also compares predicted and true adjacency weights using Spearman correlation and off-diagonal mean absolute error.Source: §F.2 · p. 22

F.3 ADDITIONAL RESULTS

The additional graph results show strong edge-weight ranking on weighted gridworlds, while exposing a precision–recall trade-off in one reconstruction head. The source code is stated to become publicly available on GitHub upon publication.

F.3 ADDITIONAL RESULTS

  • On gridworld, edge-weight rankings reach Spearman correlations of about 0.76 over true edges and 0.93 over all vertex pairs after ceiling rescaling.The medians network has recall near one but low precision on n = 20 families, while the degree oracle keeps precision and recall balanced.Source: §F.3 · p. 22

G SOURCE CODE

  • The paper states that its JAX source code will be publicly available on GitHub upon publication.Source: G SOURCE CODE · p. 22

Related papers

Loading 2610.19899v1…