PaperlayerView exact revision on arXiv

Source-linked AI summary

Kernel Autoresearch for Open-Ended Model Discovery

Richard Cornelius Suwandi, Feng Yin, Kevin Murphy

arXiv:2610.10394v1cs.LG

TL;DR

Automated kernel design must balance the broad search enabled by unrestricted programs against the validity guarantees of fixed kernel grammars. KERNAUT combines agent-written components with PSD-preserving construction contracts and a quality-diversity archive. Its discovered kernels transfer to unseen tasks, and a human-refined kernel further reduces held-out CRPS by 5.7%.

  • Problem

    Fixed kernel grammars guarantee validity but restrict search to their building blocks, while unrestricted programs do not guarantee valid kernels.

    Source: §1 · p. 1

  • Method

    KERNAUT has coding agents propose kernel components, a trusted interpreter assemble them under PSD-preserving contracts, and an archive retain high-performing programs with distinct behavior.

    Source: §1 · p. 1, §3.1 · p. 3, §3.3 · p. 4

  • Results

    Discovered kernels transferred to unseen tasks, and human refinement of DWF further reduced held-out CRPS by 5.7%.

    Source: §1 · p. 2, §4.3.1 · p. 7

  • Takeaways & Limitations

    The short discovered kernel programs encode inductive biases that researchers can interpret and refine.

    Source: §1 · p. 2, §5 · p. 9

    The validity guarantee as a kernel function requires component purity; batch-dependent components may yield PSD matrices per call without defining one kernel.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Kernels encode the inductive bias of a wide range of machine learning models, yet automated kernel design faces a fundamental dilemma. A fixed grammar of base kernels and operators guarantees validity but limits the search to structures expressible by those building blocks. Conversely, unrestricted programs remove this limitation but no longer guarantee validity. In our stress tests, 22-58% of LLM-generated kernels that pass numerical checks on random inputs fail when evaluated at different scales or dimensions. We propose Kernel Autoresearch (Kernaut), which treats kernel design as open-ended model discovery. Coding agents write kernels as programs, while construction contracts ensure that every accepted kernel is valid. A quality-diversity archive retains high-performing kernels with distinct behaviors, and novelty screening steers agents toward functionally new candidates. Our experiments demonstrate that the discovered kernels encode reusable inductive biases that generalize to unseen tasks. On held-out black-box optimization families, a discovered kernel outperforms a meta-learned deep kernel trained on the same episodes. Furthermore, kernels discovered from ten enzyme-kinetic rate laws achieve lower error than tuned ARD and deep kernel baselines on five unseen mechanisms. The discovered kernels are also interpretable programs that human researchers can refine: a human-refined version of one further reduces the held-out predictive error by 5.7% and optimization regret by 7.8%.

1 Introduction

Automated kernel design trades validity for breadth: fixed grammars guarantee valid kernels but restrict their structures, while unrestricted programs can move beyond those grammars without ensuring validity. KERNAUT uses construction contracts and a quality-diversity archive to support open-ended discovery of valid, diverse kernels.

1 Introduction

  • Fixed kernel grammars guarantee validity by construction but limit search to structures expressible through their base kernels and operators.Unrestricted program synthesis expands the search space but introduces the challenge of verifying validity.Source: §1 · p. 1
  • KERNAUT uses PSD-preserving construction contracts so a trusted interpreter can assemble valid kernels from agent-written components.The archive retains high-performing programs with distinct kernel behavior and guides later proposals.Source: §1 · p. 1
  • Agents can propose feature maps, spectral representations, input transformations, and compositions beyond a fixed grammar, while accepted programs are verified and retained for reuse across unseen tasks.The paper also analyzes short kernel programs and shows that human researchers can interpret and refine them.Source: §1 · p. 2

2 Related Work

Prior kernel-design approaches preserve validity by restricting search to predefined architectures, spectral families, or compositional grammars. KERNAUT instead applies coding-agent search to kernels while verifying each program’s construction.

2 Related Work

  • Deep, spectral, and compositional kernel methods preserve validity by restricting search to a selected architecture, spectral family, or grammar.Examples include greedy composition (Duvenaud et al., 2013; Lloyd et al., 2014), Bayesian optimization, differentiable composition (Sun et al., 2018), and LLM-guided evolution (Suwandi et al., 2025b).Source: §2 · p. 2
  • Yun et al. (2026) screen generated kernels with Cholesky factorization on sampled inputs, but 22–58% of programs passing initial checks failed at other scales or dimensions.KERNAUT instead constructs accepted kernels through PSD-preserving contracts, with validity following under the contracts’ stated assumptions.Source: §2 · p. 2
  • Coding-agent systems use propose–evaluate–revise workflows for algorithms and scientific tasks, and KERNAUT applies this pattern to kernel discovery with construction verification.Examples include FunSearch and AlphaEvolve (Novikov et al., 2025).Source: §2 · p. 2

3 Kernel Autoresearch Framework

KERNAUT lets agents propose kernel components while a trusted interpreter assembles accepted kernels under PSD-preserving contracts. Deterministic checks, a quality-diversity archive, and novelty screening support discovery of valid programs with distinct behavior.

3.1 Kernels as Programs

  • The four contracts let agents propose feature maps, spectral components, pullbacks, or closure trees that the interpreter assembles into kernels.Closure trees combine library kernels or previously accepted kernels by sums, products, and nonnegative scalar multiplication.Source: §3.1 · p. 3
  • Contracts preserve PSD when components return finite outputs of required dimensions, and validity as a kernel function additionally requires purity on the input domain.Purity rules out batch-dependent components that can produce PSD matrices per call without defining one kernel function.Source: §3.1 · p. 3

3.2 Acceptance Checks

  • Acceptance checks progress from execution and dimensions to sampled PSD tests and contract consistency, with only Tier 2 programs admitted to the archive.Backend checks also test finiteness and purity, including consistency under batch permutation, subsetting, extension, and point duplication.Source: §3.2 · pp. 3–4

3.3 Discovery Campaign

  • Campaigns use a run budget to propose new roots or revise archived programs, while a multidimensional MAP-Elites archive retains the highest-scoring program in each behavior cell.The archive’s elites are local representatives rather than necessarily the overall best program; backend tuning uses meta-training episodes.Source: §3.3 · p. 4, §3.2 · p. 4

3.4 Novelty Screening

  • Novelty screening compares centered, unit-norm Gram-matrix fingerprints against standard, multiscale, and previously discovered kernels, with a soft near-duplicate threshold.Programs below the threshold can still enter the archive if they pass acceptance checks and may later serve as inspirations.Source: §3.4 · p. 5

4 Experiments

The experiments test whether discovered kernels transfer to unseen tasks, whether open search adds value over recombining known kernels, and whether the programs encode meaningful domain structure. Across held-out benchmarks, discovered kernels transfer to new function families, mechanisms, patients, and records.

4.1 Meta-Evaluation Framework

  • The meta-evaluation freezes a program selected on new validation episodes, then tests transfer to withheld families or domain structures; selection is a finite model-selection problem.Under the stated Appendix A.5 assumptions, the selection bound depends on the number of frozen candidates, not the size of the program space.Source: §4.1 · p. 5

4.2 Baselines and Metrics

  • The experiments compare discovered kernels with fixed, learned-feature, deep or meta-learned, structure-search, and domain-specific references; predictive evaluation uses CRPS, where lower is better.Each kernel runs in the same GP with its discovered structure fixed; the backend fits only covariance amplitude and observation noise by training marginal likelihood.Source: §4.2 · pp. 5–6

4.3.1 Black-Box Optimization

  • On held-out BBO families, six of seven ensemble-selected kernels beat Matérn-5/2 in predictive CRPS, with a 3% average improvement; Astra’s gains ranged from 5.8% to 11.2%.Astra’s mean gain was 8.3% (standard deviation 2.0%, p = 0.003 against the ensemble); its average discovery matched human-refined DWF without human editing.Source: §4.3.1 · p. 6
  • The full four-contract search won 11 of 12 paired campaigns and reduced test CRPS by about 8% versus closure-only search; CKS and CAKE performed similarly to closure-only search.The independent-proposals comparison changes feedback, novelty screening, and candidate retention together, so it tests their combined contribution rather than archive sharing alone.Source: §4.3.1 · p. 7

4.3.2 Time-Series Forecasting

  • The validation-selected residual period-bank kernel ranked first among 58 frozen candidates on both CFC records, flattening toward their held-out plateaus where RBF missed the turn.It had the lowest mean validation CRPS across CO2, CH4, and N2O, and matched AutoGP, which infers a new structure for every window.Source: §4.3.2 · pp. 7–8, Figure 4 · p. 8

4.3.3 Enzyme Kinetics

  • On five held-out enzyme mechanisms, validation-selected kernels achieved about 25% lower error than Matérn; Astra’s selected kernel reduced error on every mechanism and nearly halved error versus optimized ARD.Astra’s kernel outperformed all six learned references in paired tests and combined substrate-response shapes, local substrate sensitivities, and enzyme-loading modulation.Source: §4.3.3 · p. 8

4.3.4 Glucose Dynamics

  • Against relative-time ARD Matérn, the five glucose discoveries reduced geometric-mean normalized MSE by 38–67% with two training episodes and 0–23% with six.With six episodes, frontier F1, frontier F2, and ensemble c2 reduced normalized MSE by 15–23%; the validation-selected ensemble encoded delayed meal response and an insulin effect lasting about two hours.Source: §4.3.4 · pp. 8–9

4.3.5 Takeaways

  • Discovered kernels transferred across held-out BBO families, enzyme mechanisms, glucose patients, and greenhouse-gas records; open search reduced BBO test CRPS by about 8% versus closure-only search.Interpretable programs also provided useful starting points for human refinement; replacing DWF’s joint construction with a sum of its warp and fold kernels further reduced held-out CRPS by 5.7%.Source: §4.3.5 · p. 9

5 Conclusion

KERNAUT uses coding agents and PSD-preserving construction contracts to discover valid kernels beyond a fixed library, with inductive biases that transfer to unseen tasks. The appendix details the validity guarantee, approximation power, implementation checks, and scope boundaries.

5 Conclusion

  • KERNAUT’s accepted kernels encode inductive biases that transfer to unseen tasks, and their short programs can be interpreted and refined by researchers.The conclusion also identifies possible future contracts for state-space, multi-output, structured-input, and semantic kernels.Source: §5 · p. 9

Appendix Guide

  • The appendix covers the validity proof and implementation, BBO and forecasting evaluations, DWF analysis, scientific-structure discovery, transfer, scope, and agent prompts.The method appendix states that mathematical results use exact arithmetic, with computational limits specified separately.Source: Appendix Guide · p. 13

A.1 PSD Preservation Under Trusted Assembly

  • Purity makes the interpreter’s symmetric PSD matrices Gram matrices of one kernel function across the full domain.The proof requires finite outputs of declared shape for the components; closure preserves PSD under addition, nonnegative scaling, and entrywise products.Source: §A.1 · p. 13
  • The guarantee certifies assembly under stated component assumptions, while sampled-batch checks assess conformance rather than full-domain purity.The guarantee assumes purity over the full domain.Source: §A.1 · p. 14

A.2 Approximation Power of the Feature Contract

  • The feature contract can approximate any continuous PSD kernel on a compact domain using polynomial feature maps with rational coefficients.The construction is admitted mathematically in exact arithmetic; the approximation proof controls feature-coordinate and inner-product error.Source: §A.2 · p. 14
  • The approximation result is existential: degree, dimension, and evaluation cost may grow with accuracy, with no guarantee under runtime caps or floating-point arithmetic.It does not provide an efficient synthesis algorithm or a complete verifier for unrestricted programs.Source: §A.2 · p. 14
  • Arbitrary feature maps can represent nonstationary kernels, including the linear kernel, which no combination of stationary kernels uniformly approximates on [0, 1].The passage contrasts this broader feature-map contract with a grammar that already includes a linear base kernel.Source: §A.2 · p. 14

A.3 Implementation Details

The implementation combines parameter tuning, archive-based quality-diversity search, and tiered contract checks. Validation selects among frozen programs by CRPS alone, with a finite-selection guarantee under stated assumptions.

A.3 Implementation Details

  • The scheduler selects archive elites as parents and draws inspirations across niches, while each accepted candidate occupies a cell indexed by niche, feature growth, and novelty band.Each cell retains only its best-scoring candidate; the campaign uses seven declared niches.Source: §A.3 · pp. 14–15

A.4 Evaluation and Registration Details

  • Tier 0 checks execution and shape, Tier 1 tests sampled positive semidefiniteness, and Tier 2 checks contract assembly and consistency before archive entry.The best tuned setting is checked again at Tier 2 before the program is frozen.Source: §A.3 · p. 16

A.3 Implementation Details

  • Parameter tuning evaluates the agent’s initial setting and deterministic Halton-sequence alternatives, with a fixed trial budget and meta-training data only.Defaults are 16 trials, rising to 64 for BBO and forecasting and 32 for chemistry.Source: §A.3 · p. 16
  • DWF ranks first under every tested selection weighting, and all nine re-scored programs remain archive elites.DWF also has the lowest validation CRPS among the ten frozen discoveries, at 0.540.Source: §A.3 · p. 17

A.4 Evaluation and Registration Details

  • The interpreter’s spectral convention preserves validity and reported scores, but registered descriptions can misstate a program’s period by a factor of 2π.Prompts describe frequencies in cycles per unit, while the interpreter uses angular frequency without a 2π factor.Source: §A.4 · p. 18

A.5 Soundness of Validation Selection

  • The validation bound grows with the number of candidates and shrinks with the number of validation episodes, whose evaluation can require costly black-box optimization rollouts.The guarantee also assumes scores lie in a bounded interval, while the reported evaluator does not clip CRPS.Source: §A.5 · p. 20
  • The validation guarantee applies to choosing among proposed programs, not to how often the proposer finds a strong candidate, and assumes candidate independence from validation episodes.The stated guarantee applies to meta-validation families; transfer to withheld meta-test families is evaluated empirically.Source: §A.5 · pp. 20–21

A.6 Randomized Screening Stress Test

The screening stress test finds that passing finite numerical checks does not establish kernel validity across dimensions, scales, and batch relations. On BBO benchmarks, discovered kernels transfer to held-out tasks, while repeated campaigns show predictive gains over a fixed Matérn reference and optimization results that vary by campaign.

A.6 Randomized Screening Stress Test

  • 12 of 35 unrestricted programs that passed the public Tier-1 screen were falsified by the hidden suite in the first run.The failures included non-PSD matrices, kernel-consistency violations, and unseen-shape errors.Source: §A.6 · p. 22, Figure 7 · p. 22
  • Requiring a PSD argument before coding reduced failures to 8 of 37, but did not eliminate invalid programs; trusted assembly closed the gap in the reported live search.The reduction was significant against the second run alone (Fisher exact p = 0.003) and both screen runs pooled (p = 0.02).Source: §A.6 · p. 22
  • Across three live campaigns allowing unrestricted kernel programs, all 35 accepted programs used construction contracts, and six errored on 16-dimensional inputs outside the benchmark maximum.The three validation-selected kernels achieved test CRPS values of 0.581, 0.604, and 0.610, each below the fixed Matérn-5/2 reference at 0.612.Source: §A.6 · p. 23

B.2 Full Frozen-Benchmark Results

  • Validation selected DWF before testing, and DWF had the lowest test CRPS among the discovered candidates, whose test CRPS ranged from 0.601 to 0.676.The range shows that acceptance and validation selection do not guarantee that every discovered candidate transfers well.Source: §B.2 · p. 25, Table 6 · p. 25

B.3 Meta-Learned Deep-Kernel Surrogates

  • With the same meta-training episodes, FSBO trailed DWF on both metrics in both modes; with 124 times more data, it reached parity but did not surpass DWF.At the matched budget, native FSBO scored CRPS 0.6615 against DWF’s 0.6011, and regret AUC 0.7807 against 0.6712; rich-budget native FSBO scored 0.6208 and 0.6566, respectively.Source: §B.3 · p. 26, Figure 8 · p. 26

B.4 Repeated Discovery and Search-Budget Controls

  • Across seven ensemble campaigns, mean test CRPS was 0.5930 ± 0.0153 versus 0.6117 for fixed Matérn-5/2, while regret AUC was 0.6869 ± 0.0444 versus 0.6931.Six of seven campaigns improved on the Matérn reference for CRPS, but four of seven did so for regret AUC.Source: §B.4 · p. 27, Figure 9 · p. 27
  • Under approximately matched LLM-call budgets, CAKE reached test CRPS 0.616 and regret AUC 0.855, both worse than DWF.The CAKE comparison retained its BIC objective and proposal procedure.Source: §B.4 · p. 27

B.5 Search-Ablation: Archive Sharing and Open-Ended Synthesis

The ablation compares the full archive method with independent proposals and a closure-only grammar under matched campaign budgets and evaluation. The full archive improves test CRPS over both alternatives.

B.5 Search-Ablation: Archive Sharing and Open-Ended Synthesis

  • The independent-proposal arm jointly removes feedback, cross-iteration novelty screening, and candidate retention, so its contrast tests their combined contribution.The closure arm restricts construction to the closure contract while retaining archive search.Source: §B.5 · pp. 27–28
  • All arms use the same proposer ensemble, per-campaign iteration budget, evidence ladder, and black-box optimization evaluation protocol.Twelve seeded campaigns per arm use the frozen protocol and score each validation-selected candidate once on shared meta-test tasks.Source: §B.5 · pp. 27–28, Table 7 · p. 28
  • 11 of 12 campaigns favor the full archive over both independent proposals (∆= −0.046) and closure grammar (∆= −0.047); each sign-flip test gives p = 0.006.Holm correction across the two contrasts gives p = 0.013.Source: §B.5 · p. 28
  • The full-archive mean is 0.571 across 12 validation-selected kernels, distinct from the original frozen DWF result of 0.601 and the seven-campaign mean of 0.593.This harness optimizes predictive CRPS alone, unlike the reported campaigns’ combined CRPS-and-regret score.Source: §B.5 · p. 28

B.6 A Frontier Proposer: GPT-6 Astra

The section tests a frontier coding model against the ensemble proposer with the rest of the discovery and evaluation setup held fixed. The frontier proposer improves test CRPS, though regret gains are less consistent and forecasting is an exception.

B.6 A Frontier Proposer: GPT-6 Astra

  • The comparison replaces the ensemble with GPT-6 Astra while keeping the task context, run limits, contracts, checks, tuning, validation selection, and frozen evaluation fixed.Seven BBO campaigns used 100–112 LLM calls each, compared with 101 for the first ensemble campaign.Source: §B.6 · p. 28
  • All seven frontier campaigns achieve lower test CRPS than fixed Matérn, and the frontier-versus-ensemble campaign-level difference is significant (p = 0.003).Regret AUC improves less consistently, with no statistically separated campaign-level difference (p = 0.22).Source: §B.6 · p. 28
  • The frontier archive reaches the best tuned baseline’s level after one discovery run, while the ensemble takes about eight; frontier validation CRPS keeps falling through the final run.Figure 10 shows best validation CRPS among accepted programs; thick lines average seven campaigns per proposer group.Source: §B.6 · p. 28, Figure 10 · p. 29
  • The validation-CRPS leader’s validation regret AUC falls on average for both proposer groups, but individual campaigns move in both directions.The plotted leader is the best-validation-CRPS program accepted so far, carried forward when a run accepts none.Source: §B.6 · p. 28, Figure 11 · p. 29
  • In ChemBench, frontier selection F2 reaches 0.312 test CRPS, 44% below the ensemble selection and significantly better than all six learned references.Two frontier campaigns submitted programs in only four of eight runs after proposals exceeding twelve tunable parameters were rejected and rounds ran out.Source: §B.6 · p. 28
  • In GlucoseBench both frontier campaigns beat the strongest tuned reference by 22%, versus 15% for the best ensemble campaign; forecasting is the exception.Frontier forecasting selections score 0.191–0.198 three-record CRPS, near fixed RBF at 0.205 and worse than the ensemble selection at 0.091; validation scores barely separate them.Source: §B.6 · pp. 28–29

B.7 Broad Benchmark Outside the Search Families

The appendix analyzes discovered DWF’s geometry and properties, then tests its performance beyond the search families. The broad benchmark supports it as a readable prior for approximate central reflection, with benefits varying by task symmetry and domain.

B.7 Broad Benchmark Outside the Search Families

  • Among 88 synthetic function instances, frozen DWF wins 44 with median CRPS change −0.2% (p = 0.77), while regret AUC is better on 52 with median gain 2.7% (p = 0.004).Refitting raises predictive wins to 57 and median gain to 0.6% (p = 0.025), but eight large losses yield mean change −11.9%; refit regret is not separated (p = 0.70). On real data, archived DWF wins 4 of 16 comparisons with median change −4.2% (p = 0.011); refitting wins 11 of 16 with median gain 1.3%, not significant (p = 0.12).Source: §B.7 · p. 30, Figure 12 · p. 30
  • Frozen DWF’s CRPS gains track approximate reflection symmetry: it wins 16 of 20 instances with S ≥ 0.9, while its gain correlates with S across all instances (ρ = 0.37, p = 0.0004).On 44 off-center variants, it wins 15 and has median change −3.3% (p = 0.016).Source: §B.7 · p. 30

C.1 Paired Predictive Comparisons

  • The paired predictive comparisons use 50 validation and 60 test tasks; Holm correction over five competitors within each split leaves every 0.05-level conclusion unchanged.The Wilcoxon signed-rank location-shift interpretation assumes symmetric task-level differences and complements, rather than measures, variation across independent campaigns.Source: §C.1 · p. 31, Table 8 · p. 31

C.2 Discovery Record and Frozen Geometry

  • DWF applies a monotone warp and triangular fold to each coordinate before a Matérn-5/2 kernel with lengthscale 0.5.On [0, 1], the warp differs from identity by at most 0.027; the fold peaks at 0.513 and maps the mirror pair 0.2 and 0.825 together, while the warp keeps their joint covariance separated.Source: §C.2 · p. 31, Figure 13 · p. 32

C.3 How DWF Was Proposed; C.4 Reference Implementation

  • The DWF agent submitted its formulation in its first turn and selected the top trial among 63 Halton points, while the base lengthscale remained fixed at 0.5.The submitted transform supplies pointwise features to a trusted interpreter, which applies the certified Matérn-5/2 kernel to assemble the Gram matrix without the agent building one.Source: §C.3 · pp. 31–32, §C.4 · p. 32

C.5 Prior Samples

  • Figure 14 presents prior functions from the frozen kernel before any data are seen; panels (a)–(c) share standard normals so their differences arise from the kernels.The comparison includes Matérn-5/2, frozen joint DWF, and the additive refinement.Source: §C.5 · p. 33, Figure 14 · p. 33

C.6 Regularity of DWF

  • At an interior fold, nonzero fold amplitude prevents a two-sided mean-square derivative, although one-sided derivatives exist.The result concerns prior one-sided mean-square derivatives and does not require every posterior mean or covariance section to show a kink; the argument extends across a coordinate fold in multiple dimensions.Source: §C.6 · pp. 33–34

C.7 Additive Kernels and Soft Reflection Symmetry

  • The additive refinement combines warp and fold covariance components, encouraging agreement for inputs sharing a fold value without requiring identical predictions.Under the stated compactness and universality assumptions, the additive kernel remains universal; the proposition compares regularization geometry rather than asserting an unconditional approximation advantage.Source: §C.7 · p. 34

C.8 DWF versus Parametric Input Warping

This section retunes parametric input-warping alternatives and compares them with DWF under a matched predictive objective. The selected additive refinement improves both reported means, with statistical support for CRPS but not for the regret reduction after multiplicity adjustment.

C.8 DWF versus Parametric Input Warping

  • The original power and exponential warps score 0.566–0.585 validation CRPS versus 0.540 for DWF, but use a negative log-likelihood objective, motivating retuning on a common predictive objective.Source: §C.8 · p. 34
  • The matched tuning gives seven arms 64 configurations for each of five scrambled-Halton seeds, using the same ten training episodes, GP amplitude/noise grids, and predictive CRPS objective.Each repeat’s lowest-training-CRPS configuration is retained; 50 validation episodes select a repeat before evaluation on 60 shared test episodes and eight-step BO rollouts.Source: §C.8 · p. 35
  • Test CRPS falls from 0.6011 to 0.5668 (5.7%), and regret AUC falls from 0.6712 to 0.6190 (7.8%) with the selected additive kernel.The regret reduction is descriptive because its multiplicity-adjusted test does not cross 0.05; CRPS improves on four of six family means, with losses on Bukin and Drop-Wave.Source: §C.8 · p. 35, Table 9 · p. 35
  • The additive-minus-joint CRPS difference is −0.0343, with a 95% bootstrap interval [−0.0468, −0.0211] and Holm-adjusted p = 0.00028.The paired Wilcoxon comparison adjusts across six non-anchor comparisons against joint DWF and describes differences within evaluated benchmark families.Source: §C.8 · p. 35
  • All five additive repeats select the same shared initial geometry, equal mixture weights, and branch lengthscales 0.5, yielding zero standard deviation across repeats.Joint DWF improves on either separately retuned branch, but the additive construction performs best under this protocol.Source: §C.8 · pp. 34–35

C.8.1 Sharpness Ablation

The sharpness ablation finds that smoothing preserves the additive kernel’s CRPS gain, while the forecasting appendix specifies its time-series evaluation protocol and search workflow.

C.8.1 Sharpness Ablation

  • The sharpness ablation compares sharp and smoothed joint and additive constructions, plus identity, warp-only, and fold-only controls, on shared test episodes.Each arm receives 64 configurations for each of five tuner seeds; one repeat per arm is selected on 50 validation episodes and tested on 60 episodes and eight-step BO rollouts.Source: §C.8.1 · p. 35, Table 10 · p. 36
  • The additive kernel retains a 5.7% CRPS advantage over the joint kernel after smoothing; the AUC gain is treated as descriptive.The CRPS difference is −0.03431, with adjusted p = 0.00030; the original prior has a derivative jump, but the sharp cusp is unnecessary for the observed gain.Source: Table 10 · p. 36, §C.8.1 · p. 36

D Time-Series Forecasting

  • The forecasting appendix specifies a frozen-kernel benchmark on greenhouse-gas records, with CFC-12 and CFC-11 as additional held-out records.Episodes train on a sampled prefix and evaluate the next 48 months, optionally reversing the record; training targets receive Gaussian noise and both splits are standardized using clean training-window statistics.Source: D Time-Series Forecasting · p. 36, §D.1 · p. 36

D.1 Protocol

  • Forecasting fitness is negative mean held-out CRPS, with candidate parameters tuned through declared backend spaces and a 10^-3 log(1 + p) complexity penalty.The penalty is constant across a program’s trials and therefore does not affect which parameter setting is frozen.Source: §D.1 · p. 36
  • Forecasting runs devoted 45.9% of tool calls to drafting and verification, versus 26.8% for BBO.The phase shares alone do not establish why forecasting needed more verification attempts.Source: Figure 15 · p. 37, §D.1 · p. 37
  • The fixed-reference grid includes normalized-time kernel families and fitted input-warping, spectral-mixture, and random-Fourier-feature references.The strongest grid point per family is selected using training fitness only.Source: §D.1 · p. 37
  • The forecasting search used twelve runs capped at ten rounds and 100 tool calls, producing nine Tier 2 verified programs.The runs consumed 498 backend parameter-tuning trials; three unsubmitted runs exhausted their budgets on interface-conformance failures.Source: §D.1 · p. 37

D.2 Full Time-Series Benchmark Results

The time-series results test whether frozen kernels transfer across greenhouse-gas records, including held-out records with a rise-to-decline pattern. The discovery transfers well to the CFC records on the 48-month benchmark, but its advantage does not extend to a separate ten-year forward diagnostic.

D.2 Full Time-Series Benchmark Results

  • On the original ten-window SF6 evaluation, the validation-selected kernel has CRPS 0.059 versus 0.029 for RBF; its main benefit is transfer to the CFC records.The 30-window diagnostic instead gives ARD-RBF the lowest overall CRPS, while harmonic–trend root is slightly better on forward windows.Source: §D.2 · p. 38, Table 13 · p. 39
  • CAKE scores 0.082 CRPS on validation and 0.036 on its original ten-window SF6 evaluation, compared with 0.029 for RBF.The three-record comparison is identified as the primary transfer result.Source: §D.2 · p. 38

D.3 Additional Held-Out Forecasting Records

  • On both held-out CFC records, the validation-selected discovery ranks first of 58 by mean CRPS, but it loses to the fixed kernel and CAKE on SF6.Its lower means against input warping and CAKE on the CFC records are not separated from zero; within-family score ranges are wide.Source: §D.3 · p. 39, Table 14 · p. 39
  • Across forward windows, the discovery’s equal-record mean is 0.034, versus 0.022 for CAKE and 0.024 for the best fixed kernel per record; its overall advantage comes mainly from reversed windows.On reversed windows, its equal-record mean is 0.136, versus 0.194 for AutoGP, 0.246 for CAKE, and 0.291 for RBF.Source: §D.3 · p. 39, Figure 16 · p. 40
  • AutoGP has mean CRPS 0.120 across the three records and lowers mean CRPS relative to fixed RBF, CAKE, and validation-selected depth-three CKS.AutoGP infers a new structure for each training window; the 90 windows overlap within records, so tests describe variation over these benchmark windows.Source: Table 14 · p. 39, §D.3 · p. 40
  • On ten-year forward prediction, three diagnostic splits favor CAKE over the selected discovery, whose corresponding CRPS values are 0.117, 0.079, and 0.150.The discovery’s gains on the 48-month benchmark do not establish an advantage at this longer forward horizon.Source: §D.3 · p. 42, Figure 18 · p. 42
  • For the CFC-12 peak-origin split, the Matérn and low-frequency sinusoid-bank components jointly represent the trajectory, while the fitted constant component is negligible.The sinusoid bank acts as a slow trend rather than an annual seasonal component; the split CRPS of 0.070 is an interpretive example, not an aggregate claim.Source: §D.3 · p. 42, Figure 19 · p. 43

E Scientific Structure Discovery

  • The enzyme-kinetics and glucose-dynamics benchmarks test frozen kernels on episodes from unseen mechanisms or patients, using the construction contracts unchanged.Search and validation data are disjoint from test data.Source: E Scientific Structure Discovery · p. 43

E.1 Enzyme Kinetics (ChemBench)

ChemBench evaluates whether kernels discovered from ten enzyme-kinetics domains transfer to five structurally distinct unseen mechanisms. The strongest frontier selection beats the matched references on those mechanisms, while the broader campaign results vary substantially.

E.1 Enzyme Kinetics (ChemBench)

  • ChemBench maps seven inputs to reaction rate and tests on five structurally distinct mechanisms withheld from search.The ten training domains include canonical saturation, inhibition, temperature, pH, and bisubstrate mechanisms; the evaluator accesses only the oracle’s public run interface.Source: §E.1 · p. 43
  • Across nine campaigns, the mean improvement over all nine is 8.13%, with a sample SD of 10.90 percentage points and a range from −10.6% to 23.0%.Validation and test CRPS agree closely across campaigns (Spearman ρ = 0.95, p < 10^-4); the five campaigns with lowest validation CRPS improve over the strongest evaluated fixed reference by 9.5%–23.0%.Source: §E.1 · pp. 44–45
  • Frontier F2 is significantly better than all six references and wins on at least 71 of 75 held-out episodes against each.Its improvement over Matérn-5/2 is 38%, 80%, 87%, 70%, and 27% on the five ordered mechanisms, respectively.Source: §E.1 · pp. 45–46, Table 16 · p. 46
  • E1 has the lowest mean test CRPS among ensemble campaigns and references, but is not significantly better than optimized ARD, coordinate-aware DKL, or the domain library after Holm correction.The domain library is researcher-defined and informed by the domain, rather than an automatic discovery.Source: §E.1 · p. 45
  • The F2 feature map combines ten substrate-response shapes and their local log-substrate sensitivities with enzyme, temperature, and pH factors, plus a linear backbone.Its eleven parameters are tuned by Halton search, and the feature map uses the seven public inputs rather than reading the oracle’s rate laws.Source: §E.1 · p. 46
  • On a held-out fractal-kinetics slice, the discovery follows the near-flat region and subsequent rise more closely than fixed Matérn, but deviates at high concentrations.These local curves illustrate predictive behavior; aggregate held-out performance is measured across episodes.Source: §E.1 · p. 46

E.2 Glucose Dynamics (GlucoseBench)

  • The glucose benchmark selects a kernel using child and adolescent profiles, then freezes it for testing on adult profiles and sealed interventions.Forecasting remains within each patient, while kernel selection pools information across profiles; the fixed-intervention results are not directly comparable to Murphy’s actively designed interventions and per-patient ODE fits.Source: §E.2 · p. 48

F Transfer and Interpretation of Discovered Kernels

  • The appendix frames its remaining material around transfer across domains and interpretation of discovered kernels.The supplied passage states this scope without reporting a specific transfer result.Source: F Transfer and Interpretation of… · p. 51

F.1 Cross-Domain Transfer of Selected Kernels

Frozen kernels transfer unevenly across domains: forecasting kernels help selected target settings, while chemistry kernels’ biochemical structure does not carry over under coordinate copying. The following sections describe how discoveries are reported and diagnosed through coordinate sweeps.

F.1 Cross-Domain Transfer of Selected Kernels

  • Forecasting-selected kernels approach the best fixed kernel on forecasting records but do not match the forecasting-discovered kernel; their BBO gain is smaller, and neither selection improves ChemBench.The chemistry kernel trails fixed references on BBO and forecasting under a fixed coordinate conversion, which is treated as a compatibility check rather than semantic transfer.Source: §F.1 · p. 51

F.2 Natural-Language Descriptions of Discovered Kernels

  • Discovery reports register a formulation before tuning, then check its claims against the frozen program because registration describes intent rather than tuned behavior.Table 18 distinguishes the frozen-kernel description from what tuning settled.Source: §F.2 · pp. 51–52, Table 18 · p. 52

F.3 Coordinate-Sweep Diagnostics

  • Coordinate sweeps show E1 depends mainly on enzyme loading and substrates, while F2 also responds mildly to product, inhibitor, and temperature, with little pH sensitivity in either.The sweep measures normalized kernel-feature similarity at sampled inputs; values near 1 do not establish invariance over the full domain.Source: Figure 28 · p. 53, §F.3 · p. 53

G Scope and Future Extensions

The experiments focus on small-data BBO, while the framework’s contracts support extensions to larger datasets, new kernel structures, and other applications. The section also identifies limits in current evaluation and proposes follow-up tests.

Broader problem settings

  • BBO covers at most eight input dimensions with n = 8 + 2d observations per episode; six live-search programs sized parameter arrays for eight coordinates and errored at d = 16.DWF runs unchanged as dimension increases, and exposing d to every component could prevent this implementation error.Source: G Scope and Future Extensions · p. 54

Larger datasets

  • Explicit feature and spectral contracts reduce exact inference from O(n^3) to O(nm^2) for m features, while inducing-point approximations could similarly help pullback and closure contracts.The current BBO benchmark represents the small-data regime where kernel choice matters most.Source: G Scope and Future Extensions · p. 54

New construction contracts

  • New contracts can extend the program space without changing the search loop, including multi-output kernels, structured inputs, and continuous spectral measures.The proposed multi-output construction assembles a PSD kernel from a factor and scalar kernel; graph and sequence embeddings and trusted closed-form spectral densities are also described.Source: G Scope and Future Extensions · p. 54

State-space kernels from linear ODEs

  • For stable linear ODEs driven by white noise, an interpreter could check stability, solve a Lyapunov equation, and assemble the resulting stationary Gaussian-process kernel.Agents would specify the parameterized system matrices and choose latent dimension and coupling pattern.Source: G Scope and Future Extensions · p. 54

Recurrent feature maps

  • A feature-map contract can use a fixed recurrence or ODE solver over an input’s event history, while nonlinear latent dynamics and causal world models would require a separate direction.The resulting kernel is the inner product of input-specific state features.Source: G Scope and Future Extensions · p. 54

Other kernel methods and modern applications

  • PSD-kernel applications beyond Gaussian-process covariances include kernel ridge regression, support vector machines, two-sample tests, semantic kernels, and scientific surrogates.Molecular kernels and differential-equation methods are identified as natural next domains.Source: G Scope and Future Extensions · p. 55

Selection within open-ended archives

  • Validation-selected kernels rank first among 58 frozen candidates on both unseen CFC records, but predictive mixtures of archive members are outside the reported evaluation.The experiments select one frozen program by validation CRPS rather than using a test-time Pareto frontier or averaging archive members.Source: G Scope and Future Extensions · p. 55

Protocol integrity and reward hacking

  • Proposed follow-up tests include adversarial evaluator attacks and held-out probes, plus feature anonymization to test whether discoveries use variable semantics or numerical search alone.The current backend computes Gram matrices and fit statistics, uses meta-training episodes for search scores, and freezes reported programs before testing.Source: G Scope and Future Extensions · pp. 54–55

H LLM Prompts

This appendix documents Kernaut’s agent interface and prompt templates for kernel discovery across optimization, forecasting, enzyme-kinetics, and CGM tasks. It describes candidate submission and trusted verification, task-specific search requirements and constraints, and cautions against interpreting a covariance kernel as a recovered domain model.

H.1 Agent Interface

  • The initial conversation has one user message and no fixed few-shot bank; later parent and inspiration examples are drawn from archive history and include candidate or mathematical-form metadata.Root runs have no parent payload, and inspirations remain empty only while the archive is empty.Source: §H.1 · p. 55
  • Tool calls use schema-constrained JSON, while staging supplies candidate identity, contract, source, and formulation; trusted tools return verification and evaluation results rather than relying on agent claims.The closure contract serializes sums, products, and nonnegative scaling as nested dictionaries, with documented library leaves including linear, RBF, Matérn, periodic, and spectral-mixture kernels.Source: §H.1 · pp. 55–56

H.2 Shared Prompts

  • The shared system prompt defines exact candidate contracts and entry points, and requires certified code to be deterministic, stateless, and limited to NumPy or math imports.The trusted interpreter builds kernel matrices for feature-map and transform contracts; unverified kernel-matrix candidates are for diagnostics only.Source: §H.2 · p. 56
  • The shared prompt directs agents to submit, verify, optimize declared parameters, and evaluate accepted candidates before revising behavior based on structured failures.Mathematical-novelty mode calls for distinct roots, genuine behavioral revisions, archive queries to avoid duplicates, and comparison of the quality/cost frontier.Source: §H.2 · pp. 56–57

H.3 Domain Task Descriptions

  • The procedural meta-BO prompt targets dimension-agnostic kernels for normalized tasks with varied function types, hidden coordinate transformations, and irrelevant dimensions, while excluding ordinary closure-tree rediscoveries.It prioritizes stability at dimensions 2, 5, 6, and 8 and asks agents to balance global smoothness with periodic or localized variation.Source: §H.3 · p. 58
  • Both domain prompts require agents to formulate kernels explicitly, justify positive semidefiniteness, and distinguish candidates from known kernels before implementation.The meta-BO instructions also require at least two substantially different constructions and archive checks before revisions.Source: §H.3 · pp. 58–59
  • The forecasting prompt seeks a transferable prior for long monthly series with normalized one-dimensional month inputs, small observation noise, and stability across input dimensions 1–8.It rejects discovered closure trees and asks candidates to state a PSD argument, behavioral distinction, and falsification test before staging and evaluation.Source: §H.3 · p. 59
  • Candidate code should return only certified transforms, features, or spectral components for the trusted interpreter to use, with the enzyme-kinetics prompt preferring these constructions.Source: §H.3 · pp. 59–60
  • Both domain prompts direct agents to develop at least two structurally different proposals before refining the strongest one.Source: §H.3 · pp. 59–61
  • Enzyme-kinetics kernels must transfer across datasets with unknown rate laws, using seven-dimensional inputs that encode substrate, inhibitor, product, enzyme, temperature, and pH conditions.The prompt specifies that inputs are scaled to [0, 1], with temperature spanning 278–368 K and pH spanning 4–10.Source: §H.3 · p. 59
  • Discovered enzyme-kinetics kernels must go beyond sums and products of standard kernels, and a covariance must not be presented as a recovered rate law.Source: §H.3 · pp. 59–60
  • The prompts require an exact mathematical formulation, a positive-semidefiniteness argument, a closest known kernel, a behavioral distinction, and a falsification test before candidates are staged, verified, tuned, and submitted.Source: §H.3 · p. 60
  • The CGM task targets forecasting simulated type-1 diabetes patients across six-hour episodes with one meal, at most one insulin bolus, and readings every five minutes.Each episode begins from the same physiological state, so interventions and sensor noise distinguish episodes.Source: §H.3 · p. 60
  • CGM inputs combine reading time with meal and bolus conditions; because meal and bolus coordinates are constant within an episode, the kernel governs both within-trajectory time correlations and sharing across interventions.The input is five-dimensional, with coordinates scaled to [0, 1].Source: §H.3 · pp. 60–61
  • CGM candidates must pursue mathematical novelty beyond ordinary sums and products of standard kernels, and a covariance must not be described as a recovered physiological model.Source: §H.3 · pp. 60–61

Related papers

Loading 2610.10394v1…