PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic Distributional Regression: Conditional Density Estimation through Agent-Written Programs

Lorenzo Tedesco

arXiv:2610.37611v1stat.MEstat.ML

TL;DR

Conditional density estimation describes response features beyond the mean, while choosing a useful model structure can demand substantial specification. ADR has a language-model agent write a compact conditional-distribution program; the paper establishes statistical guarantees under assumptions and reports accurate estimates in its examples.

  • Problem

    Conditional density estimation can describe spread, skewness, tail behaviour and multimodality beyond the conditional mean, but parametric models depend on adequate specification while flexible nonparametric fits may be hard to summarize and interpret.

    Source: §1 · pp. 1–2

  • Method

    ADR has a language-model agent write a short executable program that returns a conditional distribution with explicit covariate-dependent parameters and fitted constants.

    Source: §2.1 · p. 5

  • Results

    The paper establishes a finite-sample Hellinger bound, an independent-validation oracle inequality, and asymptotic normality of fitted constants for a stably selected regular structure with sufficiently accurate fitting.

    Source: §3.3 · pp. 12–14

  • Takeaways & Limitations

    The selected regular structure need not contain the true density for the fitted constants to converge to its pseudo-true value and, with sufficiently accurate fitting, be asymptotically normal.

    Source: §3.2 · p. 11, §3.3 · p. 14

    The asymptotic result assumes stable selection of a fixed regular structure and sufficiently accurate fitting, excluding unidentified or singular mixtures, active parameter constraints and irregular estimated thresholds.

    Source: §3.2 · p. 11, §3.3 · p. 14

Abstract

from arXiv · show

In this paper we propose a method for estimating conditional densities from an iid sample using a language-model agent. The agent writes the estimator as a short program that, for each covariate value, returns the estimated conditional distribution of the response. The program is built from a library of probability distributions, so the returned law can be a parametric distribution, a mixture or a multivariate factorization, with parameters given by explicit functions of the covariates. Since the number of operations and constants in the program is limited, the fitted conditional density is a formula that can be inspected and evaluated independently of the agent, while the choice of the model structure and of the fitting procedure is left to the agent. We call this approach agentic distributional regression (ADR). Treating the returned program as a random conditional density estimator, we derive a finite-sample error bound under explicit complexity and optimization assumptions, and an oracle inequality for selection using independent validation data. For a stably selected regular structure with sufficiently accurate fitting, we obtain asymptotic normality of the fitted constants. In simulation studies, on twelve real datasets and on a new dataset collected after the agents' training data, ADR with Claude Opus 5.5 has the best average performance among the methods considered.

1 Introduction

Conditional density estimation describes features of the response beyond its mean, but existing approaches trade flexibility against interpretability or require analysts to specify the model. ADR uses an agent to choose a compact conditional-distribution program, with statistical guarantees under stated assumptions and evaluations across simulations and real datasets.

1 Introduction

  • A conditional density captures spread, skewness, tail behaviour and multimodality, supporting conditional probabilities, quantiles and simulated responses beyond the conditional mean.These features matter in applications such as risk assessment and prediction intervals.Source: §1 · p. 1

1.1 Related work

  • Parametric methods retain interpretable formulas but depend on adequate specification, while flexible nonparametric estimators often lack a simple expression for inspecting covariate effects.The paper also reviews limitations of smoothing, partitioning, neural-network, quantile-regression and automated model-search approaches.Source: §1 · p. 2, §1.1 · pp. 2–3

1.2 Contribution

  • ADR has a language-model agent write a short executable conditional-distribution program whose structure and fitting procedure are chosen for each dataset.The program uses a library of probability distributions and can express location–scale regression, piecewise laws, covariate-weighted mixtures or multivariate factorizations.Source: §1.2 · pp. 3–4
  • The bounded program retains an explicit, inspectable law that supports sampling and, for univariate responses, distribution and quantile functions without needing observations at prediction time.Its shared fitted law yields ordered quantiles and makes covariate effects and regime changes readable.Source: §1.2 · p. 4
  • The analysis gives a finite-sample Hellinger bound, an independent-validation oracle inequality, and asymptotic normality for sufficiently well-fitted constants in a stably selected regular structure.The bound separates approximation, complexity and agent error; asymptotic normality does not require the selected structure to be correctly specified.Source: §1.2 · p. 4
  • The evaluation uses simulations, a real-data benchmark and Wikipedia activity data from after the agents’ training period; reported results indicate ADR can compare favourably with existing estimators, including on small samples.The study uses Claude Sonnet 5.5 and Claude Opus 5.5 in simulations, and Claude Opus 5.5 in real-data studies.Source: §1.2 · p. 4
  • Unlike BoxLM, ADR targets conditional densities with point-estimated constants and uses a compact saved formula evaluated under an audited benchmark protocol, including post-training data.One agent conducts the search without a prescribed proposal–criticism loop or dataset description.Source: §1.1 · p. 3, §1.2 · p. 4

2 Agentic distributional regression

ADR represents an estimated conditional law as a short Python program whose parameters are explicit functions of covariates. The agent can propose and revise model structures and fit constants within an execution budget, while the saved program defines reproducible predictions.

2.1 Conditional distributions as programs

  • The distribution library supports multivariate normals, products of conditionally independent distributions, sequential bivariate factorizations and mixtures.Sequential models factor the response law into a first response distribution and a second conditioned on the first response and covariates.Source: §2.1 · p. 5
  • A saved Python script defines a deterministic function that maps each covariate vector to a distribution with explicit parameter expressions and stores all fitted constants.The original observations are not needed to make predictions from the script.Source: §2.1 · p. 5
  • Available operations express nonlinear or piecewise dependence of distribution parameters and mixture weights on covariates, and on the first response in sequential models.Valid components and parameters are required for normalized laws; numerical safeguards do not guarantee smoothness, finite moments or validity over the entire covariate space.Source: §2.1 · p. 5

2.2 Output contract and fitting procedure

  • The agent receives the observations, distribution library, held-out negative-log-likelihood scoring criterion and execution budget, and may propose structures, fit constants, inspect diagnostics and revise its model.No particular optimizer or model-selection strategy is prescribed; the agent chooses both structure and fitting procedure.Source: §2.2 · pp. 5–6
  • The output contract constrains the returned expression, while the final saved script, including any refitted constants, defines the estimator; noncompliant submissions are recorded as invalid.The contract applies to the returned estimator rather than the computations used to obtain it.Source: §2.2 · pp. 6–7, Table 3 · p. 7
  • Predictions from a saved program are reproducible without rerunning the agent, although repeated estimation runs on the same data may return different structures and constants.The procedure itself is stochastic even with the same agent configuration and execution budget.Source: §2.2 · p. 6

3 Theoretical properties

This section establishes conditional-density error guarantees for agent-written programs under explicit complexity, optimization, and validation assumptions, and derives parameter limits when a regular structure is stably selected and accurately fitted. The guarantees distinguish adaptive training-sample search from independent validation and do not establish that the agent finds the structure or achieves full nonparametric efficiency.

3.1 Statistical setup

  • An ADR program has a discrete structure specifying distributions and expressions, plus fitted constants; the experimental contract limits the constant count to 80.Structures and constants are treated separately, and asymptotic analysis can allow program-node and constant-count bounds to grow with sample size.Source: §3.1 · p. 8

3.2 Assumptions

  • The adaptive-search guarantee assumes a controlled envelope class and small agent regret; neither statistical complexity nor regret follows from the program contract or time budget.Program-size limits alone do not control constant ranges or precision, and admissibility does not ensure approximation or consistency.Source: §3.2 · pp. 9–12
  • The parameter limit requires a fixed regular submodel selected with probability tending to one; it may be misspecified and excludes singular mixtures, active constraints, and irregular thresholds.Program-contract compliance alone does not establish these regularity conditions.Source: §3.2 · p. 11

3.3 Main results

  • Theorem 1 bounds Hellinger error by approximation, agent regret, comparator fluctuation, and whole-class complexity, yielding consistency when these terms vanish.Under additional moment or density-ratio conditions, it also gives integrated conditional Wasserstein or Kullback–Leibler bounds; the complexity term covers every program that may be selected.Source: §3.3 · p. 12
  • With finite comparator log-likelihood-ratio variance, the finite-variance corollary retains consistency when approximation, regret, envelope, complexity, and variance terms vanish.It replaces the comparator’s exponential-moment condition with finite variance, at the cost of a variance-dependent term.Source: §3.3 · p. 13
  • Independent validation selects a candidate whose risk is near the best of M candidates, with a penalty depending on candidate count and validation size, not their search procedure.The guarantee does not establish that the best candidate is accurate, and does not cover refitting candidates on validation data or adaptively proposing structures from validation scores.Source: §3.3 · pp. 13–14
  • Under stable selection of a fixed regular structure, the saved constants converge to its pseudo-true value and are asymptotically normal when fitting error is sufficiently small.The normal limit matches the exact likelihood minimizer’s first-order limit when ηn = oP(n^-1).Source: §3.3 · p. 14
  • With η_n = O_P(n^-1), the parameter rate is n^-1/2; asymptotic normality requires the stronger η_n = o_P(n^-1) and has sandwich covariance H^-1JH^-1.The limit θ* minimizes risk within the selected structure, so p_θ* need not equal p_0.Source: §3.3 · p. 14
  • Assumption 4 supplies high-probability selection of s*, but the theorem does not establish that the agent finds this structure or that the estimator is efficient in the full nonparametric model.The asymptotic result concerns the selected parametric structure.Source: §3.3 · p. 14

4 Simulation studies

The simulations compare ADR agents with CPFN and other estimators on univariate and bivariate conditional distributions. Opus 5.5 performs strongly across both designs, while the results also show sensitivity to model structure and the available distribution library.

4.1 Univariate responses

  • AWD was 0.076 for ADR: Opus 5.5 and 0.080 for CPFN at n = 250, and 0.042 versus 0.051 at n = 1000; Opus had the lowest mean AWD at both sizes.Opus also had the lowest AQE at reported levels except the median, where CPFN was lower at both sizes.Source: Table 4 · p. 16, §4.1 · p. 16
  • At x = 0.4 and x = 0.8, the Opus density is visually close to truth despite differing in form and parameters; CPFN is shifted at 0.4 and fills the gap between modes at 0.8.These illustrations concern one replicate, while the tables summarize replicated experiments.Source: §4.1 · p. 17, Figure 2 · p. 17
  • In the univariate example, the saved Opus program is a two-normal mixture with a shared decreasing scale and a quadratic branch split above an estimated knot at 0.353.Its AWD of 0.027 was the sixth smallest among 100 replicates; the fitted law is essentially unimodal near x = 0.5 and captures the change to two branches.Source: §4.1 · p. 17

4.2 Bivariate responses

  • AWD was 0.193 for ADR: Opus 5.5, 0.278 for CPFN, 0.684 for KCDE, and 0.789 for ADR: Sonnet 5.5; Opus had the smallest W1 at all three representative covariates.An independent sample from the true law had AWD 0.171; across five Opus runs on the same training sample, median AWD was 0.199 and mean AWD was 0.342.Source: Table 5 · p. 19, §4.2 · p. 19
  • Across five Opus runs on the same bivariate training sample, three recovered the ring and blobs, one was close, and one represented nested rings without blobs, with AWD 0.865.The five-run median AWD was 0.199, versus 0.278 for CPFN; the one-sample comparison therefore does not describe run-to-run variability.Source: §4.2 · p. 19, Table 5 · p. 19
  • The bivariate Opus program approximates the ring with a sequential factorization because the library has no annulus family; its fitted blob covariance is nearly isotropic rather than matching the true rotating covariance.The program recovers several covariate-dependent quantities, while the covariance is a principal simplification where the data or library provide less information.Source: §4.2 · pp. 19, 21, 22

5 Real-data studies

The real-data studies evaluate ADR on twelve UCI datasets and a newly collected Wikipedia dataset, using held-out negative log-likelihood. Opus 5.5 leads by mean rank on the UCI benchmark and has the smallest NLL on every Wikipedia fold, while dataset-level UCI winners vary.

5.1 The UCI benchmark

  • The UCI evaluation uses five contiguous folds, and the target is the conditional distribution of the released response, including added noise.Estimating the response before rounding and noise addition would be a different problem.Source: §5.1 · p. 22
  • ADR: Opus 5.5 had the best mean NLL on five UCI datasets, while CPFN led on three, KCDE on two, and MDN on two; Opus had the best mean rank, 2.083.The relative performance depends on the dataset.Source: §5.1 · p. 23, Table 8 · p. 23
  • For forestfires, all five fitted programs omit covariates, selecting an unconditional mixture after tried covariate effects did not reliably improve cross-validated loss.The mixture combines a near-zero normal component with a log-normal component for positive burned areas.Source: §5.1 · p. 24
  • The UCI datasets are public and old, and short selected formulas may behave poorly outside observed covariate ranges or fit recording-process features rather than scientifically relevant variation.The authors recommend independent validity checks and a scoring criterion suited to the object being estimated.Source: §5.1 · pp. 24–25

5.2 A dataset collected after the agents’ training

  • ADR had the smallest Wikipedia-dataset NLL on each fold: mean 2.495, compared with 2.739 for KCDE and 3.030 for CPFN.The modeled responses were page views and edits from randomly sampled Wikipedia articles, observed after the agents’ training data.Source: §5.2 · p. 25, Table 9 · p. 25

6 Discussion

ADR represents conditional density estimators as short executable programs, with examples illustrating accurate estimates and inspectable modeling choices. Its theoretical guarantees and empirical findings are qualified by explicit assumptions and evaluation limits.

6 Discussion

  • ADR returns a fitted conditional distribution as a short executable program, spanning parametric regression, mixtures, piecewise laws, and sequential multivariate models.Once fitted, the program can be evaluated and inspected independently of the agent.Source: §6 · pp. 25–26
  • The examples include accurate estimates for branching univariate and bivariate ring–blob designs, and show selected structures such as heavy-tailed or unconditional mixtures.The forestfires example selects an unconditional mixture after finding no reliable covariate effect; toxicity uses a heavy-tailed distribution with covariate-dependent scale.Source: §6 · p. 26
  • Theoretical guarantees require complexity and optimization conditions that are not implied by limits on program constants, syntax, or runtime.A fixed contract may also leave nonvanishing approximation error for general conditional distributions.Source: §6 · p. 26
  • Further experiments are needed to assess stability: the bivariate study used one training sample, and the real-data comparison depends on released folds, added noise, and baseline tuning.Repeated agent runs and larger prespecified simulation collections are suggested.Source: §6 · p. 26
  • Proposed follow-up work includes verifiable search guarantees, broader covariate-domain checks, and study of how computing resources and longer budgets affect discovered structures.Without domain checks, a program may apply a function such as logarithm outside the training range; the replication package includes retained programs, transcripts, and scoring scripts.Source: §6 · pp. 26–27

Appendix

The appendix collects proofs and supplementary empirical and audit material.

Appendix

  • Appendix A contains proofs of Theorems 1–3 and Corollary 1, while Appendix B provides exact-quantile simulation errors, agent instructions, and audit, repeated-run, and resource summaries.The audit procedure and repeated runs are listed under Appendix B.4.Source: Appendix · p. 31

A Proofs of the theoretical results

The proof of Theorem 1 builds a uniform likelihood-ratio bound over program-class densities by controlling a finite collection of upper envelopes and transferring the result to candidate densities.

A Proofs of the theoretical results

  • The proof first bounds likelihood ratios over a finite collection of upper envelopes, then transfers the envelope bound to every density in the program class.It next controls the comparator’s empirical loss and combines the bounds.Source: A Proofs of the theoretical results · pp. 31–32
  • Independence factors the affinity calculation across observations; Markov’s inequality and a union bound control violations across the envelopes.The number of envelopes enters through Hn(u) = log Nn(u), with tδ = log(2/δ).Source: A Proofs of the theoretical results · pp. 31–32
  • The envelope-to-density step uses the triangle inequality in L2(µ) and a common-null-set condition so the simultaneous bound remains valid after data-dependent selection.Substitution yields the bound simultaneously for all p ∈ Pn.Source: A Proofs of the theoretical results · p. 32

4. The comparator and the selected program. Let

These proof passages establish validation-selection and consistency results under stated conditions, and outline bounds for Wasserstein and Kullback–Leibler distances.

4. The comparator and the selected program. Let

  • If the quantities in the consistency assumption tend to zero, the bound vanishes and the selected density converges to the target in squared Hellinger distance in probability.The passage states this as h2(bpn, p0) → 0 in probability.Source: §4 · p. 33
  • The proof obtains the stated Wasserstein bound by coupling common probability mass identically and integrating over the covariate distribution.The passage also identifies the Kullback–Leibler conclusion as following from deterministic inequalities proved for Theorem 1.Source: §4 · pp. 33–34
  • Validation selection is controlled by simultaneous concentration over candidates and an agent choice whose validation loss is within εv of the minimum, yielding an oracle inequality.The candidate set is conditioned on before validation observations are treated as random.Source: §4 · pp. 34, 36
  • For a regular structure, uniform convergence and a uniquely minimized population risk localize the exact empirical fit, while a sufficiently accurate approximate fit converges to the same parameter.The proof proceeds through uniform convergence and localization, a Taylor expansion, and control of the difference between exact and agent-fitted constants.Source: §4 · pp. 36–37

2. Expansion of the exact likelihood fit. Let

Under the stated regularity conditions, the exact likelihood fit has parameter- and risk-rate conclusions, and its fitted constants are asymptotically normal when the fitting error is sufficiently small.

2. Expansion of the exact likelihood fit. Let

  • With probability tending to one, condition (R2) makes the empirical Hessian locally strongly convex, so the exact minimizer is interior.Its smallest eigenvalue is at least λ/2 in a sufficiently small neighborhood of θ∗.Source: §2 · p. 37
  • The expansion proves the parameter-rate statement, and substituting it proves the risk-rate statement.The argument uses η_n = o_P(n^-1) for the fitting error.Source: §2 · p. 38
  • The multivariate central limit theorem for ψ(Z_i) yields asymptotic normality; conditioning on G_n leaves the limit unchanged because Pr(G_n) → 1.Source: §2 · p. 38

B.1 Univariate errors with exact quantiles

The exact-quantile check finds only slight changes in most errors, while median AQE remains a poor accuracy indicator in this design. The agent instructions constrain the returned program and score it on held-out negative log-likelihood.

B.1 Univariate errors with exact quantiles

  • For Opus 5.5, mean AWD is 0.073 for n = 250 and 0.037 for n = 1000, versus 0.076 and 0.042 in Table 4.With exact quantiles, AWD and non-median AQE decrease slightly for both agents; Table 10 values are not comparable with other Table 4 rows.Source: §B.1 · p. 39, Table 10 · p. 39
  • At x ≥ 0.7, small departures from equal mixture weights can move the fitted median into a low-density tail, making median AQE a poor accuracy indicator.The true density at the median is less than 3% of its value at the modes there; unimodal fits place the median near the centre.Source: §B.1 · p. 39

B.2 Agent instructions

  • The agent returns a distribution-valued model for each covariate row, judged by held-out mean negative log-likelihood on the original response scale.The task contract limits programs to 80 constants and 500 AST nodes, with no loops or comprehensions.Source: §B.2 · p. 40
  • Runs have a 60-minute budget and must deliver a contract-passing model plus notes reporting validation and expected NLL.For bivariate tasks the output is a joint distribution and the criterion is held-out joint negative log-likelihood; benchmark tasks also require validity across the covariate-domain box.Source: §B.2 · p. 40

B.3 Repeated runs in the bivariate simulation

Repeated Opus 5.5 runs on the same bivariate training sample usually recovered a similar ring-and-blobs structure, but another run selected nested rings without blobs. This variation shows that exploratory structure was not always corrected by validation.

B.3 Repeated runs in the bivariate simulation

  • The first Opus run and repetitions 1 and 2 used a covariate-dependent ring-and-two-blobs mixture, with AWD values differing by at most 0.013.The table covers five Opus runs on the same training sample; Figure 4 shows draws from each program at three representative covariates.Source: §B.3 · p. 41, Table 11 · p. 41
  • Repetition 3 kept the same components but had a less accurate ring radius and more diffuse blobs.Source: §B.3 · p. 41
  • Repetition 4 instead fitted nested rings without blobs and stopped after 18 minutes; validation did not always correct exploratory structure.The authors suggest pooled response radii can appear bimodal because the ring radius is near 0.5 or 3.5 for most covariates.Source: §B.3 · p. 41

B.4 Run audits and computational accounting

The audit reports retained-run deviations and exclusions, while noting that transcript scans cannot establish statistical independence. Resource summaries describe a single server and do not establish comparative computational efficiency.

B.4 Run audits and computational accounting

  • Retained-run audits recorded conduct deviations, but no integrity breaches; bivariate, UCI, and Wikipedia run retention differed by study.The bivariate study retained one run per agent plus four Opus repetitions; no task was repeated after an audit flag.Source: §B.4 · pp. 41, 43
  • Among 400 univariate transcripts, the library’s finite-precision scoring module appeared in 40, including 39 Opus runs; reported results exclude first-attempt success rates.There were 200 retained univariate runs per configuration; retained Opus runs with the module had mean AWD 0.075 and 0.044, versus 0.076 and 0.042 for remaining runs at n = 250 and n = 1000.Source: §B.4 · p. 41
  • Visible-text scans and contract checks cannot rule out prior knowledge or establish how the agent searched, so absence of detected violations does not establish statistical independence.Source: §B.4 · p. 43
  • Agent runtime and program-size summaries reflect one server; unmatched hardware and budgets prevent claims of comparative computational efficiency.Source: §B.4 · p. 43

Related papers

Loading 2610.37611v1…