PaperlayerView exact revision on arXiv

Source-linked AI summary

From Data to Program: Fast & Direct Generative Program Inference from Empirical Data

Simon Klüttermann, Xueying Ding, Leman Akoglu

arXiv:2609.35348v1cs.LGcs.AI

TL;DR

Estimating density from finite samples often requires fitting a model separately to each dataset. PRODiGI instead learns to infer an executable generative program from data in one forward pass, with fine-tuning applied to the inferred program. It achieves lower average density and score errors than existing pretrained models with multi-fold speedups, and program-space fine-tuning improves distribution match by 84%.

  • Problem

    Classical and neural density estimators are dataset-specific, requiring their parameters to be fitted separately to each dataset, typically by optimizing a likelihood-based objective.

    Source: §1 · p. 2

  • Method

    PRODiGI is pretrained on datasets paired with known programs to infer an explicit generative program in one forward pass, and fine-tunes the inferred program’s differentiable parameters while keeping model parameters fixed.

    Source: §1 · p. 2

  • Results

    Program-space fine-tuning improves distribution match by 84%; PRODiGI achieves lower average density and score errors than existing pretrained models with multi-fold speedups.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The inferred program can be executed independently of the input data and supports direct sampling, density evaluation, and score queries without further pretrained-network calls.

    Source: §3 · p. 4

    Broadening pretraining priors and refining fine-tuning strategies to improve real-world performance remain directions for future work.

    Source: §4.1 · p. 9

Abstract

from arXiv · show

Estimating probability densities from a finite set of samples typically requires dataset-specific model fitting. We introduce PRODiGI, a pretrained data-to-program model that infers an explicit, executable generative program in a single forward pass. Pretrained on synthetic datasets paired with their ground-truth programs, PRODiGI accommodates diverse generative families and data dimensionalities through template prediction and non-autoregressive program parameter decoding. Its inferred programs support direct sampling, density and score evaluation, and inspection independently of the pretrained model. We further introduce program-space fine-tuning, which refines differentiable program parameters by matching generated and empirical samples while keeping model parameters intact. Experiments show that PRODiGI achieves lower average density and score MAE than existing pretrained models, while offering multi-fold speedups over its closest competitors. Program-space fine-tuning further reduces generation MMD by 84%. By turning empirical data into explicit, reusable programs, PRODiGI introduces a new direction for fast, interpretable tabular generative modeling.

1 INTRODUCTION

PRODiGI infers an explicit generative program directly from empirical data, addressing the per-dataset fitting required by many density estimators. Its serialized templates and non-autoregressive decoder support diverse program families and dimensions, while inference and fine-tuning operate on the program rather than the pretrained model.

1 INTRODUCTION

  • Classical generative families can be expressive and interpretable, but fitting GMMs, SCMs, and Copulas may require optimization, model selection, graph search, or dependence estimation.The cited examples include EM and component selection for GMMs, causal graph search for SCMs, and dependence estimation for Copulas.Source: §1 · p. 1
  • PRODiGI maps empirical data to a generative program in one forward pass, without explicit model fitting, search, optimization, or model selection.The inferred program can be sampled, queried, and inspected independently of the pretrained model.Source: §1 · pp. 1–2
  • The approach trains on synthetic datasets paired with known programs, teaching the model to recover each dataset’s associated serialized program rather than discard it.This amortizes program inference into pretraining.Source: §1 · p. 1
  • Unlike DiScoFormer, which needs an external iterative sampler, PRODiGI outputs an explicit generative program supporting density evaluation and direct sampling.The introduction contrasts PRODiGI with TabPFN’s repeated model evaluations across features.Source: §1 · p. 2
  • Canonical serialized templates accommodate heterogeneous priors and varying dimensions, while a non-autoregressive decoder predicts the template and fills its slots in parallel.Program-space fine-tuning updates the inferred program’s differentiable parameters while leaving the model parameters intact.Source: §1 · pp. 1–2, Figure 1 · p. 2

2 RELATED WORK

Pretrained statistical models vary in what they infer and which tasks they support directly. PRODiGI produces an explicit program that can be executed independently of the input samples and pretrained network, avoiding per-dataset optimization and repeated network calls.

Pretrained Models for Generation and Estimation

  • DiScoFormer predicts point-wise densities and scores but needs iterative model calls for generation, while TabPFN samples features sequentially through repeated conditional-model calls.DDE and NNKDE train separate models for each dimensionality; NNKDE also supports direct sampling.Source: §2 · p. 3

Comparing Classical, Neural, and Pretrained Statistical Approaches

  • PRODiGI combines inference without per-dataset optimization, explicit statistical structure, and independence from both original samples and the pretrained network after construction.It predicts programs across GMM, SCM, and Copula prior families.Source: §2 · p. 3
  • A single forward pass constructs a reusable program for generation, density estimation, and score-vector estimation without further pretrained-network evaluations.The construction cost is amortized over subsequent samples and statistical queries.Source: §2 · p. 3

3 PRODiGI FOR ZERO-SHOT DATA TO PROGRAM

PRODiGI maps empirical datasets to executable generative programs by predicting program structure and reconstructing parameters. Its program-space fine-tuning adapts inferred program parameters while keeping the pretrained model weights fixed.

3 PRODiGI FOR ZERO-SHOT DATA TO PROGRAM

  • PRODiGI encodes an empirical dataset, predicts its program template, and decodes the template’s parameters into an executable program that can run independently of the dataset.The template specifies the program family and structural attributes; the decoder reconstructs the concrete program.Source: §3 · p. 4

3.1 DATA ENCODER

  • The Set Transformer encodes unordered samples into a fixed-size representation without positional encoding, making it permutation-invariant and usable for datasets of arbitrary sample size.The representation consists of K tokens of dimension h.Source: §3.1 · p. 4

3.2 PROGRAM TEMPLATE PREDICTION

  • Templates separate each program family’s structure from its concrete values, covering GMM, SCM, and Copula priors; canonical serialization maps equivalent entity orderings to the same slot layout.Training standardizes both data and generative programs, while inference transforms the inferred program back using each input feature’s empirical mean and standard deviation.Source: §3.2 · p. 4

3.3 NON-AUTOREGRESSIVE (N-AR) PROGRAM DECODER

  • The non-autoregressive decoder predicts typed template slots in parallel, while self-attention can capture dependencies such as structural relationships within an SCM.Decoding against a structurally valid, typed template avoids autoregressive error propagation and keeps outputs consistent with the selected structure.Source: §3.3 · p. 5

3.4 TRAINING

  • Training samples programs from synthetic priors, draws empirical data from each program, and trains the encoder, template predictor, and decoder end-to-end to recover the program.The decoder receives the ground-truth template during training because a wrong predicted template would make parameter-slot supervision ill-defined; inference instead uses the predicted template.Source: §3.4 · p. 5

3.5 FINE-TUNING

  • Program-space fine-tuning refines inferred program parameters while keeping the pretrained weights and program structure fixed.It uses analytical NLL for GMMs and Copulas, and sample-based MMD for SCMs, whose exact likelihood is generally intractable.Source: §3.5 · p. 6
  • Program-space fine-tuning applies across GMM, SCM, and Copula families, refining their respective mixture parameters, functional mechanisms, or marginal and dependence parameters.Restricting optimization to the inferred program’s smaller parameter space reduces gradient and optimizer memory and requires no backpropagation through the pretrained model.Source: §3.5 · p. 6

4 EXPERIMENTS

The experiments compare PRODiGI with pretrained and classical baselines on generation, density, and score estimation. PRODiGI is fastest for generation, PRODiGI-FT leads density and score estimation, and ablations examine decoding, normalization, serialization, and dimensionality.

4 EXPERIMENTS

  • Evaluation covers generation, density, and score tasks on 96 prior-generated plus 96 real-world datasets, using pretrained and classical baselines.PRODiGI was trained for 750 epochs on 54M distinct programs.Source: §4 · pp. 6–7

4.1 MAIN RESULTS

  • 84%: PRODiGI fine-tuning reduces generation MMD to 0.89 (×10−3), approaching per-dataset-fit GMM’s 0.78 at one-third of its runtime.PRODiGI generates in 0.06 seconds, approximately 8× faster than the next-fastest method.Source: §4.1 · p. 7, Figure 3 · p. 7, Table 2 · p. 8
  • Density estimation: PRODiGI-FT reduces MAE from 7.58 to 2.19 and raises Spearman correlation from 0.69 to 0.91, leading all reported metrics and average ranks.It is approximately 16× faster than TabPFN and 3× faster than GMM.Source: §4.1 · p. 8, Table 2 · p. 8
  • Score estimation: PRODiGI-FT achieves the best MAE rank (2.38) and highest mean cosine similarity (0.75), with a cosine rank close to GMM’s at roughly one-third of its runtime.TabPFN has the highest mean score MAE (55.07) and requires more than 30 minutes, about 146× longer than PRODiGI-FT.Source: §4.1 · p. 8, Table 2 · p. 8
  • On real-world data, PRODiGI variants capture density structure and marginals, and fine-tuning reduces MMD by approximately 48%; complex manifolds challenge all methods, motivating broader priors.Figure 4 shows examples from FinancialBeneficiary and FinancialTransaction datasets.Source: §4.1 · pp. 8–9, Figure 4 · p. 9

4.2 ABLATION RESULTS AND ANALYSES

  • N-AR decoding ensures valid programs and is 44× faster than AR; normalization accommodates varying scales and lowers real-world generation MMD by 68%.The ablations are conducted on 20-dimensional data.Source: §4.2 · p. 9, Figure 5 · p. 9
  • Canonical serialization resolves reconstruction ambiguity; without it, decoding loss plateaus prematurely as the model struggles to determine module sequence.Source: §4.2 · p. 9, Figure 5 · p. 9
  • As d rises, two-sample rejection increases for all methods, while PRODiGI variants scale gracefully and benefit from larger context samples, especially at high d.Source: §4.2 · p. 9, Figure 6 · p. 9

5 CONCLUSION

PRODiGI infers executable generative programs from empirical data in one forward pass, and its program-space fine-tuning improves distribution matching. The paper contrasts this approach with dataset-specific fitting and other pretrained methods, and reports code, datasets, and a checkpoint for reproducibility.

5 CONCLUSION

  • PRODiGI achieves lower average density and score errors than existing pretrained models, multi-fold speedups, and an 84% reduction in generation MMD with program-space fine-tuning.Fine-tuning adjusts inferred program parameters while keeping the pretrained backbone fixed.Source: §5 · p. 9

AI USE STATEMENT

  • The authors disclose using generative AI to interpret results, implement and debug code, and edit the paper, with code manually verified and claims reviewed.They state that they take responsibility for the final content.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The authors publish training and inference code, all datasets, and the trained PRODiGI checkpoint, with detailed baseline results in Appendix G.Source: REPRODUCIBILITY STATEMENT · p. 10

A EXTENDED RELATED WORK

  • Classical and neural density estimators generally construct a separate estimator for each dataset through fitting or data-dependent smoothing, unlike PRODiGI’s direct mapping from samples to programs.Source: A EXTENDED RELATED WORK · p. 13
  • Tabular neural generators require dataset-specific training or fine-tuning, while PRODiGI infers a program in one pass but currently supports only continuous distributions.The current formulation does not support categorical variables.Source: A EXTENDED RELATED WORK · p. 13
  • Pretrained methods differ in their direct support for generation, density estimation, and score estimation; some require iterative sampling, sequential evaluations, or additional procedures.The capability comparison distinguishes native model outputs from additional procedures; score comparisons also depend on matching noise levels and input spaces.Source: A EXTENDED RELATED WORK · pp. 13–14, Table 3 · p. 14

B DETAILS ON DATA PRIORS

PRODiGI supports three data-prior families: Gaussian Mixtures, Structural Causal Models, and Copulas.

B DETAILS ON DATA PRIORS

B.1 GAUSSIAN MIXTURES

The appendix describes how PRODiGI’s three prior families are generated and normalized, then represented as serialized programs and templates for decoding.

B.1 GAUSSIAN MIXTURES

  • GMM datasets vary in component count and dimension; component weights have minimum size and separation constraints, with remaining weight degrees of freedom sampled from a Dirichlet distribution.The described ranges extend to 10 components and 50 dimensions.Source: §B.1 · p. 14, Table 4 · p. 14

B.2 STRUCTURAL CAUSAL MODELS

  • SCM datasets use growing random causal graphs with bounded depth, random edge weights and node-specific mechanisms; the prior limits activation choices to three separable functions.Each node also receives noise, bias, and output-scale parameters.Source: §B.2 · pp. 14–15, Figure 7 · p. 15

B.3 COPULAS

  • Copula priors combine sampled feature marginals with a Gaussian copula and an LKJ-sampled correlation matrix, allowing varied marginals and nontrivial dependence.The marginal pool includes Gaussian, Uniform, Kumaraswamy, Exponential, and Student’s t distributions.Source: §B.3 · pp. 15–16, Figure 8 · p. 16

C NORMALIZATION

  • Program normalization targets zero mean and unit variance per feature; training uses normalized programs, while inference standardizes inputs and inversely transforms the inferred program to the input scale.Copula marginal standardization preserves the copula under strictly increasing affine transformations.Source: C NORMALIZATION · p. 16, §C.3 · p. 17

C.1 GAUSSIAN MIXTURES

  • GMM normalization uses global means and feature-wise standard deviations, leaving mixture weights unchanged; SCM moments are estimated from 10,000 samples and normalization preserves the causal graph.Because SCM moments are estimated, its normalized marginals are approximately, rather than exactly, zero-mean and unit-variance.Source: §C.1 · pp. 16–17, §C.2 · p. 17

D.1 WHAT IS A GENERATIVE PROGRAM?

  • A generative program specifies a family, structural configuration, and parameters; serialization encodes those fields in an ordered sequence, including concrete marginal choices or causal structure.The examples illustrate that a serialized program specifies more than the family: it includes model-specific values and structure.Source: §D.1 · p. 18, §D.2 · pp. 18–19

D.3 TEMPLATES

  • A template records a program’s family and discrete structural attributes, then defines typed slots whose values instantiate the concrete program.The template determines the number, order, semantic role, and type of decoder outputs.Source: §D.3 · pp. 19–20

D.4 TOKENIZATION

  • Tokenization separates symbolic fields, integer quantities, and real-valued parameters; canonical ordering makes each dataset correspond to one program, while set encoding treats observations as unordered.The fixed vocabulary is built from sampled GMM, SCM, and Copula programs, and its tokens map to learnable 512-dimensional vectors.Source: §D.4 · p. 20, §D.5 · p. 20, §D.6 · p. 20

E.1 DISCOFORMER

DiScoFormer is evaluated as a density- and score-estimation model trained on GMM priors, with a Langevin chain used for generation. Its density estimates are not normalized in these experiments, and its reported fine-tuning uses ten epochs per dataset.

E.1 DISCOFORMER

  • DiScoFormer learns density and score estimates from GMM priors using analytically known log densities and score functions as supervision.Source: §E.1 · p. 21
  • Because no pretrained checkpoints were provided, the evaluation trains a separate DiScoFormer model for each tested dimensionality.The dimensions are 1, 2, 5, 10, 20, 30, 40, and 50, using default hyperparameters.Source: §E.1 · p. 21
  • DiScoFormer’s density estimates were observed not to satisfy normalization, so negative log-likelihood is not meaningful for them.Source: §E.1 · p. 21
  • Generation uses a Langevin chain initialized with context points, running 32 parallel chains with burn-in and thinning.The setup discards the first 20,000 iterations and retains every tenth generated sample.Source: §E.1 · p. 21
  • The evaluation reports DiScoFormer after ten fine-tuning epochs per dataset, since the authors observed modest additional gains through approximately ten epochs.Source: §E.1 · p. 21

E.2 TABPFN

The appendix describes how TabPFN, GMM, KDE, and NNKDE produce samples and estimates, including their fitting or inference procedures and practical constraints.

E.2 TABPFN

  • TabPFN generates samples and estimates densities by repeatedly predicting features conditionally until all features have been predicted.Score vectors are estimated from density predictions using finite differences.Source: §E.2 · p. 21
  • TabPFN score estimation uses finite differences across 2 · d density predictions, with ϵ = 0.1 because smaller values yielded noise-dominated estimates.The authors selected ϵ = 0.1 after preliminary testing.Source: §E.2 · p. 21
  • TabPFN and DiScoFormer density estimates are not normalized to integrate to one, so the evaluation omits their NLL values.Source: §E.2 · p. 21

E.3 GMM

  • The GMM baseline fits each dataset with EM, selects among 1–10 components by BIC using five initializations, and computes density and score analytically.Generation samples a component according to its fitted weight, then samples from that Gaussian.Source: §E.3 · p. 22

E.4 KDE

  • KDE selects its bandwidth by five-fold cross-validation and represents the density as Gaussian kernels centered on observed data points.It generates samples by choosing an observed point uniformly and adding Gaussian noise at the selected bandwidth.Source: §E.4 · p. 22

E.5 NNKDE

  • NNKDE predicts a local covariance for each training point’s Gaussian kernel; its evaluation uses whitening and a dataset-specific model for 40 dimensions.The authors also fine-tune a single global bandwidth scale by leave-one-out NLL for 1,000 gradient-descent steps.Source: §E.5 · pp. 22–23

F.1 CASE STUDIES

Case studies show that PRODiGI can recover the components or structure of inferred programs, and that some family-label errors still yield close distributional reconstructions.

F.1 CASE STUDIES

  • Examples show close recovery of GMM components, Copula marginals, and SCM structure, alongside close approximations of the overall dataset distributions.Figure 9 illustrates a three-component 2D GMM, a 2D Copula, and a 5D SCM.Source: §F.1 · p. 24, Figure 9 · p. 24
  • About 0.13% of in-distribution examples receive an incorrect prior-family label, yet the reconstructed distribution and marginals can still closely match the input.In one example, a 2D SCM is reconstructed as a Copula; 75% of incorrect predictions occur on one-dimensional datasets.Source: §F.1 · p. 24
  • The figure reports only 1 misclassification among 744 programs and shows that this example still fits the data well.The authors suggest family overlap, rather than inherent difficulty in fitting the data, explains such errors.Source: Figure 10 · p. 25

F.2 REAL-WORLD RESULTS AND EXAMPLES

On real-world datasets, PRODiGI remains fast and competitive despite greater distribution-matching difficulty, while fine-tuning improves reported performance and broader prior coverage remains a potential need.

F.2 REAL-WORLD RESULTS AND EXAMPLES

  • The real-world evaluation uses 96 OddBench datasets, with GMM-derived density and score ground truth available for 37 datasets.The GMM ground truth uses up to 100 components; the retained datasets pass the stated two-sample test criterion.Source: §F.2 · p. 25
  • Real-world density and score estimation remain challenging: the GMM baseline’s Spearman correlation falls from 0.84 to 0.67 and its average cosine similarity from 0.73 to 0.55.The ground truth in this evaluation is derived from a GMM.Source: §F.2 · p. 25
  • On GMM-derived real-world ground truth, PRODiGI is generally below baselines modeling GMMs or trained only on GMM data, but ranks fourth-lowest in Score MAE and second-lowest in density NLL.SCM templates are disabled for density and score estimation, while Copula templates are retained despite an expected poor fit.Source: §F.2 · p. 25
  • The authors identify broader prior-family coverage as a potential route to improving real-world performance.Figure 12’s caption notes that limited prior coverage constrains performance on datasets not adequately described by available priors.Source: §F.2 · p. 27, Figure 12 · p. 28
  • PRODiGI generates real-world samples in the shortest time, while PRODiGI-FT achieves the third-lowest average MMD despite real-world MMD being about an order of magnitude higher than on prior families.Figure 12 presents the performance–inference-time trade-off; the figure caption says PRODiGI-FT consistently improves performance across tasks.Source: §F.2 · p. 25, Figure 12 · p. 28
  • Every prior family appears 14–46 times among real-world predictions, and the authors say each captures characteristics observed in real-world data.Figure 13 shows that the families are not equally prevalent.Source: §F.2 · p. 25, Figure 13 · p. 28

F.3 PRODIGI OUTPUTS

  • The appendix’s two displayed real-world program examples are compact Copula programs, although PRODiGI can generate programs with thousands of tokens.Source: §F.3 · p. 29

G DETAILED RESULTS

The detailed-results tables organize comparisons by task, metric, dataset family, and—in several cases—dimension, with separate real-world tables.

G DETAILED RESULTS

  • Distribution-match tables report MMD, ranks, runtime, and runtime ranks across dimensions for GMM, SCM, and Copula prior datasets.A separate table describes two-sample-test rejection rates for prior-family datasets.Source: Table 10 · p. 30, Table 11 · p. 31, Table 12 · p. 32
  • Prior-family density tables cover MAE, Spearman correlation, and NLL, with dimension-specific tables for the GMM and Copula subsets.NLL is not reported for TabPFN or DiScoFormer variants because their predicted densities do not integrate to one.Source: Table 14 · p. 34, Table 17 · p. 37, Table 20 · p. 40
  • Prior-family score tables report MAE and cosine similarity, including dimension-specific GMM and Copula results.Source: Table 23 · p. 43, Table 26 · p. 45, Table 27 · p. 46
  • Real-world tables cover distribution matching and rejection rates, density MAE, Spearman correlation and NLL, and score MAE and cosine similarity.Several tables also report method ranks and runtime.Source: Table 29 · p. 48, Table 30 · p. 48, Table 31 · p. 49

Related papers

Loading 2609.35348v1…